Talk to a sales representative

+1 844-822-8378
or

Write to us

How VWO calculates ‘chance to beat original’ (CTBO)

Friday, July 25, 2014

Chance to beat original is the probability that a variation will perform better than the control version of your website or mobile app. For example, a CTBO of 60% means that the variation is likely to perform better than the control 60% of the time.

Attention!

Currently, this method of calculating the CTBO is only used for Multivariate and Personalization campaigns in VWO, as they still follow the Frequentist method of calculating a winner. To determine the winning variation in A/B and Split URL Campaigns in VWO, we follow the SmartStats (Bayesian) method of calculating the chance to beat all versions.

The success or failure of a variation is calculated by comparing the results against a base version (by default, the control page). You can change the base for your campaign at any time.

base variation

Please note that VWO calculates the CTBO on the basis of campaign threshold settings you have defined in Settings > Campaign Settings as displayed in the following screenshot:

To determine CTBO of a variation, VWO calculates the following values:

  • Conversion rates for control and variation
  • Standard error for control and variation
  • CTBO of a variation

Calculate conversion rates of the control and variations

conversion rates

Mathematically, conversion rate is a binomial random variable, meaning it can have two possible values. It is represented as the variable p. In A/B testing, we calculate the value of p by observing n visitors on the webpage that is being tested. From those n visitors, we then calculate the number of conversions. The percentage value that results is the conversion rate of your website or app.

Conversion rate (p) = actual conversion/visits to the website

The percentage value is the conversion rate of your website.

For example, let us say you are running a campaign to test the new design of your landing page. The Control receives 894 visits, of which 423 converted. On the other hand, the variation received 863 visits, of which 458 converted. This implies that the conversion percentage value of the control is 47%, while that of the variation is 53%.

Calculate standard error for the control and variations

If you repeat the same test multiple times, it is likely that you will get a slightly different value of p (or conversion rate of a variation) every time. This happens because in statistical terms, we’re sampling and like every sample, there’s a range of error associated with it (because it does not cover the entire population). To avoid doing repeated tests, we can use already established statistical formulas calculate the standard error to determine how much deviation from average conversion rate (p) can be expected from the test results. The deviation is a range within which the conversion rate is usually found. Smaller the deviation, more confident you can be about estimating true conversion rate. For a given conversion rate (p) and number of trials (n), standard error is calculated as:

Standard Error (SE) = Square root of (p * (1-p) / n)

In our example, the SE for control = Square root of (.4732*(1-.473/894) = 1.67

And, the SE for variation = Square root of (.5307*(1-.5307/863) =1.70

(Please note that the percentage value are converted to decimal in the above calculations).

To get 95% range for conversion rate, multiply the standard error value by 2 (or 1.96 to be precise). You can be sure with 95% confidence that your true conversion rate lies within this range: p % ± 2 * SE.

In VWO, we calculate conversion rate range for 80%, not 95%. Hence, we multiply the SE value by 1.28.

In the example discussed above, the standard error for the control is 1.67, while that of the variation is 1.70. Next, multiply the values by 1.28. The conversion rate range thus calculated is ± 2.14% for control and ± 2.17% for the variation.

Calculate CTBO for a variation

To determine CTBO for a variation, we calculate the Z-score for control and variation. A z-score is used in statistics to model any normal distribution as a standard normal distribution.

conversion rate range

Where,

  • Pv= Conversion rate of variation
  • Pc= Conversion rate of control
  • SEv= Standard error of variation
  • SEc= Standard error of control

For our example discussed above, the z-score can be derived as follows:

Z-score = (0.53 - 0.47)/Square root of 0.01698882 +0.0166982 = 2.41605

After the z-score is calculated, VWO calculates standard normal distribution to derive the chance to beat original percentage for the variation. You can refer to the following normal distribution table to understand how CTBO is derived for a variation:

normal distribution table

As we see, for our z-score of 2.41, the normal distribution value is approximately 99%.

To make the calculation easier for you, VWO provides a campaign significance calculator. Click here to download an excel spreadsheet to help you determine the statistical significance for your campaign.

VWO does not calculate CTBO for the following scenarios:

  • The conversion rate of control is 0 or 1
  • The conversion rate of variation is 0 or 1
  • Visitors in the variation is less than the set threshold value (sample size)
  • Standard Error of either control or variation is 0
  • Note: You can edit CTBO threshold values and sample size using VWO dashboard under Settings > Campaign Settings.

Check Your Email!
We've sent a message to
yourmail@domain.com
with an activation link in it.
Just click the link, and we'll take it from there.
Can't find the mail?
Check your spam, junk or secondary inboxes.
Still can't find it? Let us know at support@vwo.com