Before you create an A/B test on your website you form a hypothesis that you want to test. It helps you design your test, set up goals you want to optimize, and approximate what kind of impact the variations will have on the goals.
In VWO, you can use the approximate impact that you care about to estimate how long your test will take to generate statistically significant results. The problem with the current calculator in VWO was that it used a fixed impact to calculate the test duration and also didn’t allow you to change any variables once your test had run for 1 week.
This could mean that sometimes the duration estimated by VWO was not reflective of the actual time needed to get results.
We have revamped our test duration calculator to make it more accessible throughout the test. It also exposes all variables that can have an impact on the time it takes for the test to reach statistical significance.
You can use this to:
- Prioritize your hypothesis by estimating how long will a test take to reach statistical significance. The test that has the potential to create a bigger impact in less amount of time can be given a higher priority over others.
- Estimate the time it will take for the test to conclude based on your criteria. We have also included an option to customize the statistical significance threshold for the test you are planning to run. This can help you estimate the time needed for the test which you would like to conclude when the probability to beat control is say 80% instead of 95% (default).
- Understand what to do when the estimated duration has passed but the test has not reached statistical significance. This calculator can present to you an extreme case scenario. You can wait until then and if you don’t get significant results in that period too, it urges you to conclude the test.
For more details on estimating time to reach statistical significance, refer to this knowledge base article.
NOTE: The duration calculator is being rolled out in phases and will soon be available in your account.
