New reporting features: test summary and cumulative charts
To aid better visualization of A/B and multivariate test reports, we recently introduced some new features in Visual Website Optimizer. Even though these are small updates, we are sure they are a step in right direction to provide intuitive understanding of test data.
In Visual Website Optimizer, you can add multiple conversion goals to getter better perspective on performance of variations. For example, we are running an A/B test on homepage where we are testing the word “a/b” v/s “multivariate” v/s “split” in the headline World’s easiest A/B testing tool to see if it makes any impact. And we are tracking five different goals in this test. With the new test summary section in reports, conversion rate can be seen for different variations (on different goals). See a screenshot below (data is fictitious in this case):
The number in parenthesis is the visitor count for the respective variation. So, in a single glimpse you can see how test variations are performing on different conversion goals.
Visual Website Optimizer has nice looking day-wise charts in the test reports which show conversion rate for variations for different days (during the time test is running). Since conversion rate can fluctuate for different days (well, weekends are usually duller than weekdays) the chart usually looks discontinuous and it is hard to find trends in it. Take a look at example of a day-wise chart:
Though it is great at reporting number of visitors, conversions and conversion rate for each day, it isn’t that good at showing trends. So, we decided to supplement it with a chart with cumulative data. That is, a chart where visitors and conversions on a particular day has visitors and conversions added for all previous days. The resultant chart (which we unsurprisingly call Cumulative Chart) produced beautiful trends which you can clearly observe. Have a look below how a cumulative chart looks like for the same data as the above chart:
What’s next for reports?
Data is useless unless you can make it reveal its deep-hidden secrets. So, we want to introduce several new visualizations in test reports. We are currently working on adding funnels into test reports, whereby you can visualize the order in which your visitors complete the goals (for different variations). You will be able to visualize and optimize funnels for different variations for your A/B and multivariate test. We are also working on adding functionality to segment test results by date, so you will be able to pick a period and crunch data only for those dates. (You can even exclude weekends or special promotion days when you know for sure that the data is skewed and biased).
Do you have other ideas for reports and charts that we can build into Visual Website Optimizer? We would love to hear them. Visualization of data is something we absolutely love to discuss! 🙂