Talk to a sales representative

+1 844-822-8378

Write to us

I created a test in which control and variation are exactly the same, why am I getting difference in results?

Posted in Common Problems on

Actually, unless difference is statistically significant, you shouldn’t bother about the results. Due to randomness in A/B testing, percentage difference in conversion rate can occur. For example, a variation may have 10% conversion rate and control 8% conversion rate just by chance. As you test more traffic, this difference should come down and ideally with infinite number of traffic, conversion rate should be exactly same.

Following are some of the important points to consider when you get different results for exactly same control and variation.

  1. In A/A test dont just look for percentage improvement, but statistical significance too, if no statistical significance, don’t read too much in results.
  2. Even if statistical significance arrives, please allow the test to run for at least 7 more days.
  3. In case you are still doubtful of the results, you can set thresholds for chance to beat original to 99%.