How are reports to be interpreted?
The detailed reports section in VWO has effectively 2 sections, graphical report and numerical data.
The graphical part has two options, Cumulative data and Day wise data. In the graphical part the X-axis denotes conversions and the Y-axis denotes the conversion percentage.
Numerical data section contains information about visits, conversions, conversion rate and error range.
- Visitors: It represents the total number of unique visitors that became part of that particular variation.
- Error range: The numbers precede by ± represents error range of conversions, here is a blogpost on how it is calculated.
- Conversion rate: It is the ratio of the number of conversions in a variation to the number of visits in the variation.
- Percentage improvement: It’s the improvement in conversion percentage values compared to control conversion percentage.
- Chance to beat original: Chance to beat original is the percentage confidence you can have in a variation outperforming Control. For example, 95% chance to beat original means we have confidence level of 95% that variation will actually outperform the control in a real world scenario. However, there is still a 5% probability that variation is not better than the control. Here is an article which explains how chance to beat original is calculated.
In addition to the above report elements, MVT has a section report for each section as shown below in the image
Impact: this paramater tells which section in your multivariate test has most influence in conversion rate. A section with higher impact has more impact on conversion rate than the one with lower impact. Note that this impact can be either positive or negative. Impact is mostly useful if you have added multiple sections in a multivariate test, and are trying to determine which one is the most important one.