{"id":142,"date":"2022-03-02T07:01:29","date_gmt":"2022-03-02T07:01:29","guid":{"rendered":"https:\/\/vwo.com\/glossary\/test-duration\/"},"modified":"2025-06-19T10:13:45","modified_gmt":"2025-06-19T10:13:45","slug":"test-duration","status":"publish","type":"post","link":"https:\/\/vwo.com\/glossary\/test-duration\/","title":{"rendered":"Test Duration"},"content":{"rendered":"\n

Prior to statistical hypothesis testing<\/a>, a lot of preplanning takes place before you collect any data. This planning includes identifying the data to gather, how to collect it, and how to measure it among many other details. A crucial part of the planning is determining how much data you need to collect<\/a> using which the overall duration of the test can be estimated.<\/p>\n\n\n\n

Why is it important?<\/strong><\/h2>\n\n\n\n

A pre-study calculation of the required sample size and test duration is performed in the majority of quantitative studies. This calculation helps to evaluate the cost, time, and resources needed to be deployed to detect an effect relevant to the study. The Minimum Detectable Effect (MDE) size is the smallest improvement that you want the experiment to detect in a metric (like conversion rate, the average revenue) associated with an existing asset. Performing a study without these estimates can be detrimental to the study. If the study is stopped before sufficient data, one may not be able to detect an important existing effect, whereas a prolonged study may waste time, resources, and money.  Therefore, it is important to determine the optimal sample size. Moreover, calculating the sample size in the design stage of the study is increasingly becoming a necessity when seeking approval for certain research projects. <\/p>\n\n\n\n

How to calculate the sample size for randomized controlled trials?<\/strong><\/h2>\n\n\n\n

Depending on the type of study design and the study’s outcome(s), formulas for the sample size calculation can vary. These calculations are generally performed during the design of randomized controlled trials (RCTs) and depending on the primary outcome of the study, a formula is chosen.<\/p>\n\n\n\n

VWO uses Bayesian metrics<\/a> to compute statistical significance<\/a> among the variations created on the platform. The duration calculator<\/a> we use thus incorporates these metrics inside its calculation and aims to determine the number of visitors it would take to reach certain metric thresholds. You can take a free trial with VWO<\/a> or request a demo from our product experts<\/a> to explore this in detail.<\/p>\n\n\n\n

Relationship of MDE on variability in sample size<\/strong><\/h2>\n\n\n\n

The greater the effect size on the outcome, the smaller the sample size needed to prove it. And conversely, to determine smaller effects, it is necessary to increase the sample size. <\/p>\n\n\n\n

The sample size calculation is based on assumptions about parameters of a test that can be inaccurate and is therefore subject to error. While choosing the estimates employed in calculating the sample size, one should be realistic. Highly optimistic choices about the effect size increase the risk of calculating an insufficient number of observations for the sample, whereas highly pessimistic choices can make the study unviable by resulting in a sample size that is too large to be collected in a practical scenario.<\/p>\n\n\n\n

How to determine the minimum effect size?<\/strong><\/h2>\n\n\n\n

The minimum effect size is the effect that would be meaningful for your business, so one would put in the effort to deploy the variation. In certain cases, deployment of a new variation can invite certain deployment costs, and the only way to justify those costs is when the proposed variation is satisfying a minimum uplift over the current design.<\/p>\n\n\n\n

For a hypothesis test, even the tiniest effect size can be found statistically significant with a large enough sample. At first, this may seem good \u2013 just collect large enough data and find statistical significance for the tiny effect and that\u2019s it. But is that statistical significance result useful for your business? Do you want to be making claims that an effect exists even if it\u2019s tiny?<\/p>\n\n\n\n

Good sample size calculations are based on the smallest effects that are meaningful for your business.<\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

The test duration comes from a sample size calculator which estimates the number of samples needed to estimate a significant effect within the chosen error bounds correctly. This calculation is performed before running an experiment to plan resources such that a timeline of an experiment is decided.<\/p>\n","protected":false},"author":565,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-142","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"acf":[],"yoast_head":"\nTest Duration | VWO<\/title>\n<meta name=\"description\" content=\"Test duration is the specific time period over which an A\/B test is run. It is imperative to determine the minimum test duration of an A\/B test for reaching a conclusive result.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/vwo.com\/glossary\/test-duration\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Test Duration | VWO\" \/>\n<meta property=\"og:description\" content=\"Test duration is the specific time period over which an A\/B test is run. It is imperative to determine the minimum test duration of an A\/B test for reaching a conclusive result.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/vwo.com\/glossary\/test-duration\/\" \/>\n<meta property=\"og:site_name\" content=\"VWO Glossary\" \/>\n<meta property=\"article:published_time\" content=\"2022-03-02T07:01:29+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-06-19T10:13:45+00:00\" \/>\n<meta name=\"author\" content=\"avnishsharamav\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/vwo.com\/glossary\/test-duration\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/vwo.com\/glossary\/test-duration\/\"},\"author\":{\"name\":\"avnishsharamav\",\"@id\":\"https:\/\/vwo.com\/glossary\/#\/schema\/person\/620cb97e975598f02bdaf7e87d218044\"},\"headline\":\"Test Duration\",\"datePublished\":\"2022-03-02T07:01:29+00:00\",\"dateModified\":\"2025-06-19T10:13:45+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/vwo.com\/glossary\/test-duration\/\"},\"wordCount\":612,\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/vwo.com\/glossary\/test-duration\/\",\"url\":\"https:\/\/vwo.com\/glossary\/test-duration\/\",\"name\":\"Test Duration | VWO\",\"isPartOf\":{\"@id\":\"https:\/\/vwo.com\/glossary\/#website\"},\"datePublished\":\"2022-03-02T07:01:29+00:00\",\"dateModified\":\"2025-06-19T10:13:45+00:00\",\"author\":{\"@id\":\"https:\/\/vwo.com\/glossary\/#\/schema\/person\/620cb97e975598f02bdaf7e87d218044\"},\"description\":\"Test duration is the specific time period over which an A\/B test is run. It is imperative to determine the minimum test duration of an A\/B test for reaching a conclusive result.\",\"breadcrumb\":{\"@id\":\"https:\/\/vwo.com\/glossary\/test-duration\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/vwo.com\/glossary\/test-duration\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/vwo.com\/glossary\/test-duration\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/vwo.com\/glossary\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Test Duration\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/vwo.com\/glossary\/#website\",\"url\":\"https:\/\/vwo.com\/glossary\/\",\"name\":\"VWO Glossary\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/vwo.com\/glossary\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/vwo.com\/glossary\/#\/schema\/person\/620cb97e975598f02bdaf7e87d218044\",\"name\":\"avnishsharamav\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/vwo.com\/glossary\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/5b76ede4c1368d6104eaa4834bd018786dcfa0a69cd759ca31da38e37a39c1e7?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/5b76ede4c1368d6104eaa4834bd018786dcfa0a69cd759ca31da38e37a39c1e7?s=96&d=mm&r=g\",\"caption\":\"avnishsharamav\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Test Duration | VWO","description":"Test duration is the specific time period over which an A\/B test is run. It is imperative to determine the minimum test duration of an A\/B test for reaching a conclusive result.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/vwo.com\/glossary\/test-duration\/","og_locale":"en_US","og_type":"article","og_title":"Test Duration | VWO","og_description":"Test duration is the specific time period over which an A\/B test is run. It is imperative to determine the minimum test duration of an A\/B test for reaching a conclusive result.","og_url":"https:\/\/vwo.com\/glossary\/test-duration\/","og_site_name":"VWO Glossary","article_published_time":"2022-03-02T07:01:29+00:00","article_modified_time":"2025-06-19T10:13:45+00:00","author":"avnishsharamav","twitter_card":"summary_large_image","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/vwo.com\/glossary\/test-duration\/#article","isPartOf":{"@id":"https:\/\/vwo.com\/glossary\/test-duration\/"},"author":{"name":"avnishsharamav","@id":"https:\/\/vwo.com\/glossary\/#\/schema\/person\/620cb97e975598f02bdaf7e87d218044"},"headline":"Test Duration","datePublished":"2022-03-02T07:01:29+00:00","dateModified":"2025-06-19T10:13:45+00:00","mainEntityOfPage":{"@id":"https:\/\/vwo.com\/glossary\/test-duration\/"},"wordCount":612,"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/vwo.com\/glossary\/test-duration\/","url":"https:\/\/vwo.com\/glossary\/test-duration\/","name":"Test Duration | VWO","isPartOf":{"@id":"https:\/\/vwo.com\/glossary\/#website"},"datePublished":"2022-03-02T07:01:29+00:00","dateModified":"2025-06-19T10:13:45+00:00","author":{"@id":"https:\/\/vwo.com\/glossary\/#\/schema\/person\/620cb97e975598f02bdaf7e87d218044"},"description":"Test duration is the specific time period over which an A\/B test is run. It is imperative to determine the minimum test duration of an A\/B test for reaching a conclusive result.","breadcrumb":{"@id":"https:\/\/vwo.com\/glossary\/test-duration\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/vwo.com\/glossary\/test-duration\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/vwo.com\/glossary\/test-duration\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/vwo.com\/glossary\/"},{"@type":"ListItem","position":2,"name":"Test Duration"}]},{"@type":"WebSite","@id":"https:\/\/vwo.com\/glossary\/#website","url":"https:\/\/vwo.com\/glossary\/","name":"VWO Glossary","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/vwo.com\/glossary\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/vwo.com\/glossary\/#\/schema\/person\/620cb97e975598f02bdaf7e87d218044","name":"avnishsharamav","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/vwo.com\/glossary\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/5b76ede4c1368d6104eaa4834bd018786dcfa0a69cd759ca31da38e37a39c1e7?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/5b76ede4c1368d6104eaa4834bd018786dcfa0a69cd759ca31da38e37a39c1e7?s=96&d=mm&r=g","caption":"avnishsharamav"}}]}},"_links":{"self":[{"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/posts\/142","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/users\/565"}],"replies":[{"embeddable":true,"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/comments?post=142"}],"version-history":[{"count":11,"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/posts\/142\/revisions"}],"predecessor-version":[{"id":4623,"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/posts\/142\/revisions\/4623"}],"wp:attachment":[{"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/media?parent=142"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/categories?post=142"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/tags?post=142"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}