{"id":3483,"date":"2012-08-22T18:50:18","date_gmt":"2012-08-22T13:20:18","guid":{"rendered":"https:\/\/vwo.com\/blog\/?p=3483"},"modified":"2025-05-01T15:46:12","modified_gmt":"2025-05-01T10:16:12","slug":"how-to-calculate-ab-test-sample-size","status":"publish","type":"post","link":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/","title":{"rendered":"How to Calculate A\/B Testing Sample Sizes?"},"content":{"rendered":"\n

(This post is a scientific explanation of the optimal sample size for your tests to hold true statistically. VWO’s test reporting is engineered in a way<\/a> that you would not waste your time looking up p-values or determining statistical significance<\/a> – the platform reports ‘probability to win’ and makes test results easy to interpret. Sign up for a free trial here<\/a>)<\/em><\/p>\n\n\n

How large does the sample size need to be?<\/strong><\/h2>\n\n\n

In the online world, the possibilities for A\/B testing<\/a> just about anything are immense. And many experiments are done<\/a> indeed, the result of which are interpreted following the rules of null-hypothesis testing, \u201care the results statistically significant<\/a>?\u201d<\/p>\n\n\n\n

An important aspect in the work of the database analyst then is to determine appropriate sample sizes<\/a> for these tests.<\/p>\n\n\n\n

Download Free: A\/B Testing Guide<\/span><\/a><\/em><\/h2>\n\n\n\n

On the basis of a daily case, a number of current approaches for calculating desired sample size<\/a> are discussed.<\/p>\n\n\n

Case for calculating sample size<\/strong><\/h3>\n\n\n

The marketer has devised an alternative for a landing page<\/a> and wants to put this alternative to a test. The original landing page has a known conversion of 4%. The expected conversion of the alternative page is 5%. So the marketer asks the analyst “how large should the sample be to demonstrate with statistical significance<\/a> that the alternative is better than the original?”<\/p>\n\n\n\n

Solution: “default sample size”<\/strong><\/h4>\n\n\n\n

The analyst says: split run (A\/B test) with 5,000 observations each and a one-sided test with a reliability of .95. Out of habit.<\/p>\n\n\n\n

What happens here?<\/em><\/p>\n\n\n\n

What happens when drawing two samples to estimate the difference between the two, with a one-sided test and a reliability of .95? This can be demonstrated by infinitely drawing two samples of 5,000 observations neach<\/span> from a population with a conversion of 4%, and plotting the difference in conversion per pair (per ‘test’) between the two samples in a chart.<\/p>\n\n\n\n

Figure 1: sampling distribution for<\/span> the difference between two proportions with p1=p2=.<\/span>04 and n1=n2=5,<\/span>000; a significance area is indicated for alpha=.<\/span>05 (reliability= .95) using a one-sided test.<\/em><\/p>\n\n\n

\n
\"Sampling<\/figure>\n<\/div>\n\n\n

This chart reflects what is formally called the \u2018sampling distribution for<\/span> the difference between two proportions.\u2019 It is the probability distribution of all possible sample results calculated for<\/span> the difference between p1=p2=.<\/span>04 andn1=n2=5,<\/span>000. This distribution is the basis \u2013the reference distribution- for null hypothesis testing<\/a>. <\/strong>The null hypothesis being that there is no difference between the two landing pages. This is the distribution used for actually deciding on significance or non-significance.<\/p>\n\n\n\n

p=.<\/span>04 means 4% conversion. Statisticians usually talk about proportions that can lie between 0 and 1, whereas in the everyday language mostly percentages are communicated. In order to comply with the chart, the proportion<\/span> notation is used.<\/p>\n\n\n\n

This probability distribution can be replicated roughly using this spss<\/span> syntax (thirty paired samples from a population.sps<\/span><\/a>). Not infinitely, but 30 times two samples are drawn with p1=p2=.<\/span>04 and n1=n2=5,<\/span>000. The difference between the two samples are then plotted in a histogram with the normal distribution inputted (the last chart in the output). This normal curve will be quite similar to the curve in figure 1. The reason for performing this experiment is to demonstrate the essence of a sampling distribution.<\/p>\n\n\n\n

The modal value of the difference in conversion between the two groups is zero. That makes sense, both groups come from the same population with a conversion of 4%. Deviations from zero both<\/span> to the left (original does better) and to the right (alternative<\/span> does better) can and will occur, just by chance. The further from zero, however, the smaller the probability of happening. The pink area with the character alpha indicated in it is the significance area, or unreliability=1-reliability=1-.<\/span>95.<\/p>\n\n\n\n

If in a test the difference in conversion between the alternative page and the original page falls in the pink area, then the null hypothesis that there is no difference between both pages is rejected in favour<\/span> of the hypothesis that the alternative page returns a higher conversion than the original. The logic behind this is that if the null hypothesis were really true, such result would be a rather \u2018rare\u2019 outcome.<\/p>\n\n\n\n

The x axis in figure 1 doesn\u2019t display the value of the test statistic (Z in this case) as would usually be the case. For clarity sake the concrete difference in conversion between the two landing pages has been displayed.<\/p>\n\n\n\n

So when in a split run test the alternative landing page returns a conversion rate that is 0.645% higher or more than the original landing page (hence falls in the significance area), then the null hypothesis stating there is no difference in conversion between the landing pages is rejected in favour<\/span> of the hypothesis<\/a> that the alternative does better (the 0.645% corresponds to a test statistic Z value of 1.65).<\/p>\n\n\n\n

Advantage<\/span> of the approach “default sample size” is that by choosing a fixed sample size, a certain standardization is brought in. Various tests are comparable \u2018stand an equal chance\u2019 to that respect.<\/p>\n\n\n\n

Disadvantage<\/span> to this approach is that whereas the chance to reject the null hypothesis when the null hypothesis (H0<\/sub>) is<\/span> true<\/span> is well known, namely the self-selected alpha of.<\/span>05, the chance to not <\/em>reject H0<\/sub>when<\/span> H0<\/sub> is<\/span> not<\/em> true remains unknown<\/em>. These are two false decisions<\/em>, known as type 1 error and type 2 error<\/a> respectively.<\/p>\n\n\n\n

A type 1 error, or alpha<\/strong>, is made when H0 <\/sub>is<\/span> rejected, when in fact H0<\/sub> is<\/span> true. Alpha is the probability of saying on the outcome of a test there is an effect for<\/span> the manipulation, while on population level there actually is none. 1-alpha is the chance to accept the null hypothesis<\/a> when it is true \u2013a correct decision<\/em>-. This is called reliability.<\/strong><\/p>\n\n\n\n

A type 2 error<\/a>, or beta<\/strong>, is made when H0 <\/sub>is<\/span> not<\/em> rejected, when in fact H0<\/sub> is<\/span> not<\/em> true. Beta is the probability of saying on the outcome of a test there is no effect for<\/span> the manipulation, while on population level there actually is. 1-beta is the chance to reject the null hypothesis when it is not true\u2013a correct decision<\/em>-. This is called power<\/strong>.<\/p>\n\n\n\n

Power is a function of alpha, sample size and effect (the effect here is the difference in conversion between the two landing pages, i.e. at<\/span> population level the added value of the alternate site compared to the original site). The smaller alpha, sample size or effect the smaller power is.<\/p>\n\n\n\n

In this example alpha is set by the analyst at.05. Sample sizes are also set by the analyst, 5000 for original, 5000 for alternative<\/span>. Which leaves the effect. And the actual effect is by definition unknown. However it is not unrealistic to use commercial targets or experiential numbers as an anchor value, as was formulated by the marketer in the current case: an expected improvement from 4% to 5%. Now if that effect were really true, the marketer of course would want to find statistically significant results in a test.<\/p>\n\n\n\n

An example may help to make this concept insightful and to clarify the importance of power: suppose the actual (=population) conversion of the alternative page is indeed 5%. The sampling distribution for<\/span> the difference between two proportions with conversion1=4%, conversion2=5% and n1=n2=5,<\/span>000is plotted in combination with the previously shown sampling distribution for<\/span> the difference between two proportions with conversion1=conversion2=4% and n1=n2=5,<\/span>000 (figure 1).<\/p>\n\n\n\n

Figure 2: sampling distributions for the difference between two proportions with p1=p2=.<\/span>04, n1=n2=5,<\/span>000(<\/span>red line) and p1=.<\/span>04, p2=.<\/span>05, n1=n2=5,<\/span>000 (dotted blue line), with a one-sided test and a reliability of .95.<\/em><\/p>\n\n\n

\n
\"Figure<\/figure>\n<\/div>\n\n\n

The dotted blue line shows the sampling distribution of the difference in conversion rates between original and alternative when in reality (on population level) the original page makes 4% conversion<\/span> and the alternate page 5%, with samples of 5,000 each. The sampling distribution whenH<\/span>0<\/sub> is<\/span> true, the red line, has basically shifted to the right. The modal value of this new distribution with the supposed effect of 1% is of course 1%, with random deviations both tothe<\/span> left and to the right.<\/p>\n\n\n\n

Now, all outcomes, i.e. test<\/span> results, on the right side of the green line (marking the significance area) are regarded as significant. All observations on the left side of the green line are regarded as not<\/em> significant. The area under the \u2018blue\u2019 distribution left of the significance line is beta, the chance to not reject H0<\/sub> when<\/span> H0<\/sub>is<\/span> in fact not true (a false decision), and it covers 22% of that distribution.<\/p>\n\n\n\n

That makes the area under the blue distribution to the right of the significance line the power area and this area covers 78% of the sampling distribution. The probability to reject H0<\/sub> when<\/span> H0<\/sub>is<\/span> not true, a correct decision.<\/p>\n\n\n\n

So the power of this specific test with its specific parameters is .78.<\/p>\n\n\n\n

In 78% of the cases when this test is done, it will yield a significant effect and consequent rejecting of H0<\/sub>. Could be acceptable, or could perhaps not be acceptable; that is a question for marketer<\/span> and analyst to agree upon.<\/p>\n\n\n\n

No simple matter, but important. Suppose for example that an expectation of 10% increase in conversion would be realistic as well as commercially interesting: 4.0% original versus 4.4% for<\/span> the alternative. Then the situation changes as follows.<\/p>\n\n\n\n

Figure 3: sampling distributions for the difference between two proportions with p1=p2=.<\/span>040, n1=n2=5,<\/span>000 (red line) and p1=.<\/span>040, p2=.<\/span>044, n1=n2=5,<\/span>000 (dotted blue line), with a one-sided test and a reliability of .95.<\/em><\/p>\n\n\n

\n
\"Figure<\/figure>\n<\/div>\n\n\n

Now the power is.26. Under these circumstances the test would not make much sense, is in fact counter-productive<\/span>, since the chance that such test will lead to a significant result is as low as .26.<\/p>\n\n\n\n

The above figures are calculated and made with the application \u2018Gpower\u2019<\/strong>:<\/p>\n\n\n\n

This program calculates achieved power <\/strong>for many types of tests, based on desired sample size, alpha, and supposed effect.<\/p>\n\n\n\n

Likewise required sample size<\/strong> can be calculated from desired power, alpha and expected effect, required alpha <\/strong>can be calculated from desired power, sample size and expected effect and required effect <\/strong>can be calculated from desired power, alpha and sample size.<\/p>\n\n\n\n

Should a power of .95 be desired for a supposed p1=.<\/span>040, p2=.<\/span>044, then the required sample sizes are 54.428 each.<\/p>\n\n\n\n

Figure 4: sampling distributions for the difference between two proportions with p1=p2=.<\/span>040 (red line) and p1=.<\/span>040, p2=.<\/span>044 (dotted blue line), using a one-sided test, with a reliability of .95 and a power of .95.<\/em><\/p>\n\n\n

\n
\"Figure<\/figure>\n<\/div>\n\n\n

This figure shows information omitted in previous charts. This also gives an impression of the interface of the program.<\/p>\n\n\n\n

Download Free: A\/B Testing Guide<\/span><\/a><\/em><\/h2>\n\n\n\n

Important aspects of power analysis<\/strong> are careful evaluation of the consequences of rejecting the null hypothesis when the null hypothesis is in fact true – e.g. based<\/span> on test results a costly campaign is implemented under the assumption that it will be a success and that success doesn\u2019t come true – and the consequences of not rejecting the null hypothesis when the null hypothesis is not true -e.g. based<\/span> on test results a campaign is not implemented, whereas it would have been a success.<\/p>\n\n\n\n

Solution: “default number of conversions” <\/strong><\/h4>\n\n\n\n

The analyst says: split run with a minimum of 100 conversions per competing page and a one-sided test with a reliability of .95.<\/p>\n\n\n\n

In the current case with expected conversion of the original page 4% and expected conversion of the alternate page 5%, a minimum of 2,500 observations per page will be advised.<\/p>\n\n\n\n

When put to the power test though, this scenario demonstrates a power of just little<\/span> over .5.<\/p>\n\n\n\n

Figure 5: sampling distributions for the difference between two proportions with p1=p2=.04, n1=n2=2500 (red line) and p1=.04, p2=.05, n1=n2=2500 (dotted blue line) rusing a one-sided test, with a reliability of .95.<\/em><\/p>\n\n\n

\n
\"Figure<\/figure>\n<\/div>\n\n\n

For a better power, a greater effect should be present, a larger sample size must be chosen, or alpha should be increased, e.g. to<\/span> .2:<\/p>\n\n\n

\n
\"Figure<\/figure>\n<\/div>\n\n\n

An alpha of .2 returns a power of .8. The power is more acceptable; the ‘cost ‘ for this bigger power consists of a magnified chance to reject H0<\/sub> when<\/span> H0 <\/sub>is<\/span> actually true.<\/p>\n\n\n\n

Again, business considerations involving the impact of alpha and beta play a key role in such decisions.<\/p>\n\n\n\n

Approach “default number of conversions” with its rule of thumb on the number of conversions actually puts a kind of limit on effect sizes that still make sense to be put to a test (i.e. with a reasonable power). In that regard it also comprises a sort of standardization and that in itself is not a problem, as long as its consequences are understood and recognized<\/span>.<\/p>\n\n\n\n

Solution: \u201csignificant sample result\u201d<\/strong><\/h4>\n\n\n\n

The analyst says: split run with enough observations to get a statistical significant result if in the test the supposed effect and actually occurs, tested one-sided with a reliability of .95.<\/p>\n\n\n\n

That sounds a little weird, and it is. Unfortunately this logic is often applied in practice. The required sample size is basically calculated assuming the supposed effect to actually occur in the sample.<\/p>\n\n\n\n

In the used example: if in a test the original has a conversion of 4% and he alternative 5%, then 2,800 cases per group would be necessary to reach statistical significance. This can be demonstrated with the accompanying spss<\/span> syntax (limit<\/span> at significant test result.sps<\/span>).<\/p>\n\n\n\n

These sort of calculations are applied by various online tools offering to calculate sample size. This approach ignores the concept of random sampling error, thus ignoring the essence of inferential statistics and null hypothesis testing. In practice, this will always yield a power of .5 plus a small additional excess.<\/p>\n\n\n\n

Figure7: sampling distributions for the difference between two proportions with p1=p2=.<\/span>04, n1=n2=2800 (red line) and p1=.<\/span>04, p2=.<\/span>05, n1=n2=2800 (dotted blue line), using a one-sided test, with a reliability of .95.<\/em><\/p>\n\n\n

\n
\"Figure7:<\/figure>\n<\/div>\n\n\n

Using this system a sort of standardisation<\/span> is actually also applied, namely on<\/span> power, but that’s not the apparent goal this method was invented for.<\/p>\n\n\n

Solution: \u201cdefault reliability and power\u201d<\/strong><\/h4>\n\n\n

The analyst says: split run with a power of .8 and a reliability of .95 with a one-sided test.<\/p>\n\n\n\n

In the current case with 4% conversion for original page versus 5% expected conversion for the alternate page, alpha=.<\/span>05 and power=.<\/span>80, Gpower advises two samples of 5313.<\/p>\n\n\n\n

Figure 8: sampling distributions for the difference between two proportions with p1=p2=.<\/span>04(<\/span>red line) and p1=.<\/span>04, p2=.<\/span>05 (dotted blue line), using a one-sided test with reliability .95 and power .80.<\/em><\/p>\n\n\n

\n
\"Figure<\/figure>\n<\/div>\n\n\n

This approach uses desired reliability, expected effect and<\/em> desired power in the calculation of the required sample size.<\/p>\n\n\n\n

Now the analyst has grip on the probability an expected\/desired\/necessary effect will lead to statistically significant results in a test, namely .8.<\/p>\n\n\n\n

Some online tools, for example VWO’s Split Test Duration Calculator<\/a>, use the concept of power in their sample size calculation.<\/p>\n\n\n\n

In a presentation by VWO “Visitors needed for A\/B testing<\/a>” a power of .8 is mentioned as a regular measure.<\/p>\n\n\n\n

It can be questioned why that should be an acceptable rule? Why could the size of the power, as well as the size of the reliability not be used more dynamically?<\/p>\n\n\n\n

Solution: “desired reliability and power\u201d<\/strong><\/h4>\n\n\n\n

The analyst says: split run with desired power and reliability using a one-sided test.<\/p>\n\n\n\n

Follows a discussion on what is acceptable power and reliability in this case, with as a conclusion, say, both 90%. Result according to Gpower, 2 times 5.645 observations:<\/p>\n\n\n\n

Figure 9: sampling distributions for the difference between two proportions with p1=p2=.<\/span>04 (red line) and p1=.<\/span>04, p2=.<\/span>05 (dotted blue line), using a one-sided test with reliability=.<\/span>90 and power=.<\/span>90.<\/em><\/p>\n\n\n

\n
\"Figure<\/figure>\n<\/div>\n\n\n

What if the marketer says “It takes too long to gather that many observations. The landing page will then not be important anymore. There is room for a total of 3,000 test observations. Reliability is equally important as power. The test should preferably be carried out and a decision should follow”?<\/p>\n\n\n\n

Result on the basis of this constraint: reliability and power both<\/span> .75. If this doesn\u2019t pose problems for those concerned, the test may continue on the basis of alpha=.<\/span>25 and power=.<\/span>75.<\/p>\n\n\n\n

Figure 10: sampling distributions for the difference between two proportions with p1=p2=.<\/span>04, n1=n2=1500 (red line), and p1=.<\/span>04, p2=.<\/span>05, n1=n2=1500(<\/span>dotted blue line), using a one-sided test with equal reliability and power.<\/em><\/p>\n\n\n

\n
\"Figure<\/figure>\n<\/div>\n\n\n

This approach allows for flexible choice of reliability and power. The consequent lack of standardization is a disadvantage.<\/p>\n\n\n

Conclusion<\/strong><\/h2>\n\n\n

There are multiple approaches to calculate the required sample size, from questionable logic to strongly substantiated.<\/p>\n\n\n\n

For strategically important \u2018crucial experiments\u2019, preference goes out to the most comprehensive method in which both “desired reliability and power” are involved in the calculation. If there is no possibility of checking against prior effects, an effect can be estimated using a pilot with “default sample size” or “default number of conversions”.<\/p>\n\n\n\n

For the majority of decisions throughout the year \u201cdefault reliability and power\u201d is recommended, for reasons of comparability between tests.<\/p>\n\n\n\n

Working with the recommended approaches based on calculated risk will lead to valuable optimization and correct decision making.<\/p>\n\n\n\n

Note: Screenshots used in the blog belong to the author.<\/em><\/p>\n\n\n

FAQs on A\/B testing sample size<\/h2>\n\n\n
What is the formula for determining sample size? <\/strong>

There are multiple approaches to determine the required sample size for A\/B testing. For strategically important \u2018crucial experiments\u2019, preference goes out to the most comprehensive method in which both \u201cdesired reliability and power\u201d are involved in the calculation.<\/p> <\/div>

What should be the required sample size for an ab test? <\/strong>

In the online world the possibilities for a\/b testing just about anything are immense. The sample size should be large enough to demonstrate with statistical significance that the alternative version is better than the original. <\/p> <\/div> <\/div>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

(This post is a scientific explanation of the optimal sample size for your tests to hold true statistically. VWO’s test reporting is engineered in a way that you would not waste your time looking up p-values or determining statistical significance – the platform reports ‘probability to win’ and makes test results easy to interpret. Sign…<\/p>\n","protected":false},"author":17,"featured_media":56979,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"post_read_time":14,"footnotes":""},"categories":[10558],"tags":[],"feature":[1852,10526],"industry-type":[],"product":[10626],"role":[10636],"region":[],"class_list":["post-3483","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-calculator","feature-ab-testing","feature-experimentation-platform"],"acf":[],"yoast_head":"\nLearn How to Calculate A\/B Testing Sample Sizes<\/title>\n<meta name=\"description\" content=\"In this detailed post, learn about the different approaches for calculating the desired sample size for creating & starting A\/B tests.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Learn How to Calculate A\/B Testing Sample Sizes\" \/>\n<meta property=\"og:description\" content=\"In this detailed post, learn about the different approaches for calculating the desired sample size for creating & starting A\/B tests.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/vwoofficial\/\" \/>\n<meta property=\"article:published_time\" content=\"2012-08-22T13:20:18+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-01T10:16:12+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2012\/08\/OG-image_How-to-Calculate-AB-Testing-Sample-Sizes.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Kees Schippers\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@VWO\" \/>\n<meta name=\"twitter:site\" content=\"@VWO\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kees Schippers\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"16 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/\"},\"author\":{\"name\":\"Kees Schippers\",\"@id\":\"https:\/\/vwo.com\/blog\/#\/schema\/person\/c430ae4e61c3fb5b63536e40ac4e6a52\"},\"headline\":\"How to Calculate A\/B Testing Sample Sizes?\",\"datePublished\":\"2012-08-22T13:20:18+00:00\",\"dateModified\":\"2025-05-01T10:16:12+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/\"},\"wordCount\":2898,\"publisher\":{\"@id\":\"https:\/\/vwo.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2012\/08\/Feature-image_How-to-Calculate-AB-Testing-Sample-Sizes.png\",\"articleSection\":[\"Calculator\"],\"inLanguage\":\"en-US\"},{\"@type\":[\"WebPage\",\"FAQPage\"],\"@id\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/\",\"url\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/\",\"name\":\"Learn How to Calculate A\/B Testing Sample Sizes\",\"isPartOf\":{\"@id\":\"https:\/\/vwo.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2012\/08\/Feature-image_How-to-Calculate-AB-Testing-Sample-Sizes.png\",\"datePublished\":\"2012-08-22T13:20:18+00:00\",\"dateModified\":\"2025-05-01T10:16:12+00:00\",\"description\":\"In this detailed post, learn about the different approaches for calculating the desired sample size for creating & starting A\/B tests.\",\"breadcrumb\":{\"@id\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#breadcrumb\"},\"mainEntity\":[{\"@id\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#faq-question-1580295142333\"},{\"@id\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#faq-question-1580295173233\"}],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#primaryimage\",\"url\":\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2012\/08\/Feature-image_How-to-Calculate-AB-Testing-Sample-Sizes.png\",\"contentUrl\":\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2012\/08\/Feature-image_How-to-Calculate-AB-Testing-Sample-Sizes.png\",\"width\":1200,\"height\":700,\"caption\":\"How To Calculate Ab Testing Sample Sizes?\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/vwo.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Calculator\",\"item\":\"https:\/\/vwo.com\/blog\/calculator\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"How to Calculate A\/B Testing Sample Sizes?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/vwo.com\/blog\/#website\",\"url\":\"https:\/\/vwo.com\/blog\/\",\"name\":\"Blog\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/vwo.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/vwo.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/vwo.com\/blog\/#organization\",\"name\":\"VWO\",\"url\":\"https:\/\/vwo.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/vwo.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2018\/09\/VWOLogo.png\",\"contentUrl\":\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2018\/09\/VWOLogo.png\",\"width\":780,\"height\":492,\"caption\":\"VWO\"},\"image\":{\"@id\":\"https:\/\/vwo.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/vwoofficial\/\",\"https:\/\/x.com\/VWO\",\"https:\/\/www.instagram.com\/vwoofficial\/\",\"https:\/\/www.linkedin.com\/company\/vwo\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/vwo.com\/blog\/#\/schema\/person\/c430ae4e61c3fb5b63536e40ac4e6a52\",\"name\":\"Kees Schippers\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/vwo.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9e9a4e152dff5ac17292f9a76fcfa21ef884b2261be21c19d830af52824d438b?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9e9a4e152dff5ac17292f9a76fcfa21ef884b2261be21c19d830af52824d438b?s=96&d=mm&r=g\",\"caption\":\"Kees Schippers\"},\"description\":\"I am a passionate Marketing Data Analyst. I offer hands on training in Statistics & Data Mining with IBM SPSS, KNIME and R.\",\"url\":\"https:\/\/vwo.com\/blog\/author\/keesschippers\/\"},{\"@type\":\"Question\",\"@id\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#faq-question-1580295142333\",\"position\":1,\"url\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#faq-question-1580295142333\",\"name\":\"What is the formula for determining sample size?\",\"answerCount\":1,\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"There are multiple approaches to determine the required sample size for A\/B testing. For strategically important \u2018crucial experiments\u2019, preference goes out to the most comprehensive method in which both \u201cdesired reliability and power\u201d are involved in the calculation.\",\"inLanguage\":\"en-US\"},\"inLanguage\":\"en-US\"},{\"@type\":\"Question\",\"@id\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#faq-question-1580295173233\",\"position\":2,\"url\":\"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#faq-question-1580295173233\",\"name\":\"What should be the required sample size for an ab test?\",\"answerCount\":1,\"acceptedAnswer\":{\"@type\":\"Answer\",\"text\":\"In the online world the possibilities for a\/b testing just about anything are immense. The sample size should be large enough to demonstrate with statistical significance that the alternative version is better than the original. \",\"inLanguage\":\"en-US\"},\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Learn How to Calculate A\/B Testing Sample Sizes","description":"In this detailed post, learn about the different approaches for calculating the desired sample size for creating & starting A\/B tests.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/","og_locale":"en_US","og_type":"article","og_title":"Learn How to Calculate A\/B Testing Sample Sizes","og_description":"In this detailed post, learn about the different approaches for calculating the desired sample size for creating & starting A\/B tests.","og_url":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/","og_site_name":"Blog","article_publisher":"https:\/\/www.facebook.com\/vwoofficial\/","article_published_time":"2012-08-22T13:20:18+00:00","article_modified_time":"2025-05-01T10:16:12+00:00","og_image":[{"width":1200,"height":630,"url":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2012\/08\/OG-image_How-to-Calculate-AB-Testing-Sample-Sizes.png","type":"image\/png"}],"author":"Kees Schippers","twitter_card":"summary_large_image","twitter_creator":"@VWO","twitter_site":"@VWO","twitter_misc":{"Written by":"Kees Schippers","Est. reading time":"16 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#article","isPartOf":{"@id":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/"},"author":{"name":"Kees Schippers","@id":"https:\/\/vwo.com\/blog\/#\/schema\/person\/c430ae4e61c3fb5b63536e40ac4e6a52"},"headline":"How to Calculate A\/B Testing Sample Sizes?","datePublished":"2012-08-22T13:20:18+00:00","dateModified":"2025-05-01T10:16:12+00:00","mainEntityOfPage":{"@id":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/"},"wordCount":2898,"publisher":{"@id":"https:\/\/vwo.com\/blog\/#organization"},"image":{"@id":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#primaryimage"},"thumbnailUrl":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2012\/08\/Feature-image_How-to-Calculate-AB-Testing-Sample-Sizes.png","articleSection":["Calculator"],"inLanguage":"en-US"},{"@type":["WebPage","FAQPage"],"@id":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/","url":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/","name":"Learn How to Calculate A\/B Testing Sample Sizes","isPartOf":{"@id":"https:\/\/vwo.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#primaryimage"},"image":{"@id":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#primaryimage"},"thumbnailUrl":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2012\/08\/Feature-image_How-to-Calculate-AB-Testing-Sample-Sizes.png","datePublished":"2012-08-22T13:20:18+00:00","dateModified":"2025-05-01T10:16:12+00:00","description":"In this detailed post, learn about the different approaches for calculating the desired sample size for creating & starting A\/B tests.","breadcrumb":{"@id":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#breadcrumb"},"mainEntity":[{"@id":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#faq-question-1580295142333"},{"@id":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#faq-question-1580295173233"}],"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#primaryimage","url":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2012\/08\/Feature-image_How-to-Calculate-AB-Testing-Sample-Sizes.png","contentUrl":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2012\/08\/Feature-image_How-to-Calculate-AB-Testing-Sample-Sizes.png","width":1200,"height":700,"caption":"How To Calculate Ab Testing Sample Sizes?"},{"@type":"BreadcrumbList","@id":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/vwo.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Calculator","item":"https:\/\/vwo.com\/blog\/calculator\/"},{"@type":"ListItem","position":3,"name":"How to Calculate A\/B Testing Sample Sizes?"}]},{"@type":"WebSite","@id":"https:\/\/vwo.com\/blog\/#website","url":"https:\/\/vwo.com\/blog\/","name":"Blog","description":"","publisher":{"@id":"https:\/\/vwo.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/vwo.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/vwo.com\/blog\/#organization","name":"VWO","url":"https:\/\/vwo.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/vwo.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2018\/09\/VWOLogo.png","contentUrl":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2018\/09\/VWOLogo.png","width":780,"height":492,"caption":"VWO"},"image":{"@id":"https:\/\/vwo.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/vwoofficial\/","https:\/\/x.com\/VWO","https:\/\/www.instagram.com\/vwoofficial\/","https:\/\/www.linkedin.com\/company\/vwo"]},{"@type":"Person","@id":"https:\/\/vwo.com\/blog\/#\/schema\/person\/c430ae4e61c3fb5b63536e40ac4e6a52","name":"Kees Schippers","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/vwo.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/9e9a4e152dff5ac17292f9a76fcfa21ef884b2261be21c19d830af52824d438b?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9e9a4e152dff5ac17292f9a76fcfa21ef884b2261be21c19d830af52824d438b?s=96&d=mm&r=g","caption":"Kees Schippers"},"description":"I am a passionate Marketing Data Analyst. I offer hands on training in Statistics & Data Mining with IBM SPSS, KNIME and R.","url":"https:\/\/vwo.com\/blog\/author\/keesschippers\/"},{"@type":"Question","@id":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#faq-question-1580295142333","position":1,"url":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#faq-question-1580295142333","name":"What is the formula for determining sample size?","answerCount":1,"acceptedAnswer":{"@type":"Answer","text":"There are multiple approaches to determine the required sample size for A\/B testing. For strategically important \u2018crucial experiments\u2019, preference goes out to the most comprehensive method in which both \u201cdesired reliability and power\u201d are involved in the calculation.","inLanguage":"en-US"},"inLanguage":"en-US"},{"@type":"Question","@id":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#faq-question-1580295173233","position":2,"url":"https:\/\/vwo.com\/blog\/how-to-calculate-ab-test-sample-size\/#faq-question-1580295173233","name":"What should be the required sample size for an ab test?","answerCount":1,"acceptedAnswer":{"@type":"Answer","text":"In the online world the possibilities for a\/b testing just about anything are immense. The sample size should be large enough to demonstrate with statistical significance that the alternative version is better than the original. ","inLanguage":"en-US"},"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/posts\/3483","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/users\/17"}],"replies":[{"embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/comments?post=3483"}],"version-history":[{"count":67,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/posts\/3483\/revisions"}],"predecessor-version":[{"id":95620,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/posts\/3483\/revisions\/95620"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/media\/56979"}],"wp:attachment":[{"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/media?parent=3483"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/categories?post=3483"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/tags?post=3483"},{"taxonomy":"feature","embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/feature?post=3483"},{"taxonomy":"industry-type","embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/industry-type?post=3483"},{"taxonomy":"product","embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/product?post=3483"},{"taxonomy":"role","embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/role?post=3483"},{"taxonomy":"region","embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/region?post=3483"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}