VWO Logo
Dashboard

VWO SmartStats for Smarter Business Decisions

VWO’s Bayesian-powered statistics engine is designed to do the heavy lifting when it comes to calculations and accuracy for you and gives you all the ingredients you need to make the right business decisions.

Know more
Complies with:
VWO GDPR Ready Badge
VWO CCPA Ready Badge
VWO G2Crowd Leader Spring Badge
Trustradius badge for VWO

We tweet useful stuff daily

Related content:

A/B Test Duration Calculator [Free Downloadable Excel]

3 Min Read

In a previous post, I provided a downloadable A/B testing significance calculator (in excel). In this post, I will provide a free calculator which lets you estimate how many days should you run a test in order to obtain statistically significant results. But, first, a disclaimer.

There is no guarantee of results for an A/B test

When someone asks how long should s/he run an A/B test, the ideal answer would be until eternity or till the time you get results (whichever is sooner). In an A/B test, you can never say with full confidence that you will get statistically significant results after running the test X number of days. Instead, what you can say is that there is 80% (or 95%, whatever you choose) probability of getting statistically significant result (if it indeed exists) after X number of days. But, of course, it may be the case that there is in fact no difference in performance of control and variation so no matter how long you wait, you will never get a statistically significant result.

So, how long should you run your A/B test?

Download and use the calculator below to find out how many visitors you need to include in the test. There are 4 pieces of information that you need to enter:

  • Conversion Rate of original page
  • What % difference in conversion rate do you want to detect (if you want to detect even the slightest improvement, it will take much longer)
  • Number of variations to test (more variations you test, more traffic you need)
  • Average daily traffic on your site (not really needed, optional)

Once you enter these 4 parameters, the calculator below will find out how many visitors you need to test (for 80% and 95% probability of finding the result). You can stop the test after you test those many visitors, but you should never stop earlier than that. You may end up concluding wrong results.

A/B test duration calculator (Excel spreadsheet)

Click below to download the calculator:

ab test duration calculator excel sheet

Download A/B testing duration calculator.

Please feel free to share the file with your friends and colleagues or post it on your blog/twitter.

By the way, if you want to do quick calculations, we have a version of this calculator hosted on Google Docs (this will make a copy of the Google sheet into your own account before you can make any changes to it).

For all the people looking to calculate their calculation without the trouble of going through “sheets or documents”, we created a simple to use A/B testing duration calculator.

How does the calculator work?

Ah! The million dollar calculator. Explaining how it works is beyond the scope of this post as it is too technical (maybe a separate post). But, if you have got stomach for it, below is gist of how we calculate number of visitors needed to get significant results.

how does the a/b testing duration calculator work?

The graph above is taken from an excellent book called Statistical Rules of Thumb[1].
Luckily, the chapter on estimating sample size is available to download freely. Another excellent source to get more information on sample size estimation for A/B testing is Microsoft’s paper: Controlled Experiments on the Web: Survey and Practical Guide[2].

Hope you like the calculator and related A/B testing tools & resources. Excited to know your thoughts at marketing@vwo.com!

More from VWO on A/B Testing
The famous Monty Hall problem and what it has to do with A/B testing

The famous Monty Hall problem and what it has to do with A/B testing

The feedback I received on “How large should your A/B test sample size be?” inspired…

Read More
Kees Schippers

Kees Schippers

7 Min Read
A/B Testing for Big Wins – When You Should Do It & How You Should Do It

A/B Testing for Big Wins – When You Should Do It & How You Should Do It

Button color tests. Font size tests. Headline tests. Run them all you want. But remember…

Read More
Smriti Chawla

Smriti Chawla

8 Min Read
Multi-Armed Bandit (MAB) – A/B Testing Sans Regret

Multi-Armed Bandit (MAB) – A/B Testing Sans Regret

Most readers of this blog would be familiar with A/B Testing. Just as a quick…

Read More
Shubhankar Gupta

Shubhankar Gupta

11 Min Read

Scale your A/B testing and experimentation with VWO.

Start Free Trial Request Demo
Shanaz Khan from VWO

Hi, I am Shanaz from the VWO Research Desk.

Join our community of 10,000+ Marketing, Product & UX Folks today & never miss the latest from the world of experience optimization.

A value for this field is required.

Thank you!

Check your inbox for the confirmation mail