A/B Split Test Significance Calculator
Built with ❤️ for testing, optimization, UX, CRO, and design teams.
P-Value
0Significant?
YesThe P-Value is x.xx Hence, your results are statistically significant!
Awesome, you understand what p-value stands for! Unfortunately most people are unable to correctly interpret p-values. Hence we built VWO SmartStats, a Bayesian statistical engine that dispenses with the need of a p-value altogether.
Unfortunately, this isn't what the p-value actually means. Don't worry, most people are unable to correctly interpret p-values. Hence we built VWO SmartStats, a Bayesian statistical engine that dispenses with the need of a p-value altogether.
P-Value
(Range from 0-1)0.334
Significance
No
Why do we use Bayesian statistics?
Intuitive Test Reports
We realized our non-statistical users frequently (and wrongly) interpreted the frequentist p-value as a Bayesian posterior probability (the probability that variation is better than control). So we built the industry's first Bayesian statistical engine that gives you an easily understandable result. An intuitive result ensures that you don't make a mistake while A/B testing revenue or other critical KPIs.
No Sample Sizing Required
VWO SmartStats relies on Bayesian inference which unlike a frequentist approach doesn’t need a minimum sample size. This allows you to run A/B tests on parts of your website or apps that might not get a lot of traffic to improve them. However, getting more traffic on your tests allows VWO to determine your conversion rates with more certainty allowing you to be more confident about your test results.
Actionable Results, Faster
VWO SmartStats was engineered keeping one key metric in mind: Speed. We have traded-off some accuracy for speed, not a lot, just a tiny bit, enough to get quicker results without impacting your bottom line. This frees up your time enabling you to test more. Also, on the off chance that you would want to be absolutely and completely sure, we calculate the maximum potential loss you'd be taking, and you can decide if the loss value matches your risk appetite.
Frequently Asked Questions
The null hypothesis states that there is no difference between the control and the variation. This essentially means that the conversion rate of the variation will be similar to the conversion rate of the control.
The p-value is defined as the probability of getting results at least as extreme as the ones you observed, given that the null hypothesis is correct, where the null hypothesis in A/B testing is that the variant and the control are the same.
Statistical significance quantifies whether a result obtained is likely due to a chance or some to some factor of interest. When a finding is significant, it essentially means you can feel confident that a difference is real, not that you just got lucky (or unlucky) in choosing the sample.
Statistical power is the probability of finding an effect when the effect is real. So a statistical power of 80% means that out of 100 tests where variations are different, 20 tests will conclude that variations are the same and no effect exists.
