Actually, unless difference is statistically significant, you shouldn’t bother about the results. Due to randomness in A/B testing, percentage difference in conversion rate can occur. For example, a variation may have 10% conversion rate and control 8% conversion rate just by chance. As you test more traffic, this difference should come down and ideally with infinite number of traffic, conversion rate should be exactly same.
Following are some of the important points to consider when you get different results for exactly same control and variation.
- In A/A test dont just look for percentage improvement, but statistical significance too, if no statistical significance, don’t read too much in results.
- Even if statistical significance arrives, please allow the test to run for at least 7 more days.
- In case you are still doubtful of the results, you can set thresholds for chance to beat original to 99%.