{"id":2260,"date":"2024-03-14T09:34:57","date_gmt":"2024-03-14T09:34:57","guid":{"rendered":"https:\/\/vwo.com\/glossary\/?p=2260"},"modified":"2025-07-08T13:08:33","modified_gmt":"2025-07-08T13:08:33","slug":"sequential-testing-correction","status":"publish","type":"post","link":"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/","title":{"rendered":"Sequential Testing Correction"},"content":{"rendered":"\n

Sequential testing is a statistical method used to analyze data as it is collected, ensuring decisions are made in a step-by-step way rather than waiting until all data is collected.<\/p>\n\n\n\n

This can help to reduce the time and resources needed for experimentation, particularly in situations where the outcome becomes clear before all data is collected. <\/p>\n\n\n\n

Let\u2019s say you’re a product manager for an eCommerce website<\/a>, and you’re planning to roll out a new feature aimed at increasing conversion rates. However, your website has limited traffic, and acquiring additional traffic through advertising campaigns is costly. In such a case, sequential testing would be ideal. <\/p>\n\n\n\n

You could implement the new feature and use sequential testing to monitor its performance. If the feature shows significant positive results early on, you can conclude the test sooner and roll out the feature to all visitors, saving time and resources. On the other hand, if the feature doesn’t perform as expected, you can stop the test early, preventing further investment of resources in an ineffective feature. <\/p>\n\n\n\n

Sequential testing correction encompasses methods aimed at preventing the issues that arise from sequential testing, such as false conclusions when interpreting interim results. Sometimes, sequential testing may heighten the risk of erroneously concluding a variation to be better when it isn’t (a false positive<\/a>). Sequential testing correction mitigates this risk by adjusting the threshold of confidence necessary before finalizing significance.<\/p>\n\n\n\n

Fixed horizon tests vs Sequential tests<\/h2>\n\n\n\n

In contrast to sequential tests, fixed horizon tests have both sample sizes and experiment goals predetermined. Conclusions can only be drawn upon completion of the review period. This approach generally provides a higher level of statistical trustworthiness but at the cost of higher traffic being used for each experiment. <\/p>\n\n\n\n

Why are Sequential tests more suited to modern A\/B testing? <\/h2>\n\n\n\n

In recent years, sequential tests have become increasingly popular, enabling continuous data collection. Here are some reasons why it is more suited to modern A\/B testing:<\/p>\n\n\n\n

Efficiency<\/h3>\n\n\n\n

By implementing sequential testing, organizations can quickly identify potential disadvantageous ideas or content at an early stage of development before they are fully implemented or exposed to a large audience. Organizations can effectively allocate resources and minimize the overall costs associated with implementing such ideas. This helps businesses make informed decisions, such as releasing a big feature before a major event, in fast-paced digital environments. <\/p>\n\n\n\n

Flexibility<\/h3>\n\n\n\n

Modern businesses need experimentation to be visitor efficient so that A\/B testing<\/a> can be done on pages with low traffic<\/a> as well. With sequential testing, sample sizes are not fixed, offering the option to stop the experiment early if significant results are observed or to continue until reaching a predetermined endpoint, accommodating varying traffic levels and experiment durations.<\/p>\n\n\n\n

What are the problems caused by sequential testing?<\/h2>\n\n\n\n

Despite having benefits, sequential testing may also pose problems for businesses. <\/p>\n\n\n\n

It may seem counterintuitive, but whenever statistical results are calculated multiple times, there is a risk of increasing the false positive rate.<\/p>\n\n\n\n

This is the main concern with continuously monitoring A\/B test statistics. Therefore, several solutions have been proposed to sequentially correct test statistics and reduce the occurrence of false positives in sequential testing<\/p>\n\n\n\n

How do you correct errors from sequential testing?<\/h2>\n\n\n\n

There are a couple of ways of correcting errors in sequential testing. They are as follows:<\/p>\n\n\n\n

Bonferroni corrections<\/h3>\n\n\n\n

False positive rates increase linearly with the number of interim checks you make. The most simple solution is to divide your false positive rate by the number of interim checks you are making.<\/p>\n\n\n\n

So, if you need a 5% false positive rate, and you are making 10 interim analyses, set the false positive rate of the test to be 5\/10 = 0.5%. This is the Bonferroni correction.<\/p>\n\n\n\n

Always valid inference <\/h3>\n\n\n\n

This method allows for continuous testing during data collection without determining in advance when to stop or how many interim analyses to conduct. This approach offers flexibility, as it doesn’t require prior knowledge of sample size and supports both streaming and batch data processing. <\/p>\n\n\n\n

Always Valid Inference isn’t popular because it’s complex to grasp and significantly compromises statistical power<\/a>. This implies that detecting a winner will take significantly longer when one actually exists.<\/p>\n\n\n\n

To simplify the testing process and allow you to focus on running tests and obtaining early results without concern for skewed outcomes, VWO uses a derivative of an approach called Alpha-Spending to correct Sequential Testing by Lan and DeMets.<\/p>\n\n\n\n

The alpha-spending approach involves distributing the type I error<\/a> (alpha) across the duration of a sequential A\/B test. With this approach, alpha can be allocated flexibly across the selected peek times, and it is only utilized when peeking occurs. If a peek is skipped, the unused alpha can be retained for future use. Additionally, there is no need to predetermine the number of tests or the timing of their execution during data collection.<\/p>\n\n\n\n

By selecting Sequential Testing Correction in the SmartStats Configuration, decision probabilities will be adjusted to minimize errors while monitoring test results during data collection in the new test reports.<\/p>\n\n\n\n

If you prioritize obtaining reliable test results and desire greater control over test statistics, consider using VWO<\/a>, where our testing system is designed to meet your advanced needs.<\/p>\n","protected":false},"excerpt":{"rendered":"

Sequential testing correction consists of techniques used in statistical analysis to mitigate the risk of increased false positives while continuously monitoring test statistics for significance. Sequential testing can increase the chance of declaring a variation to be a winner when it is actually equivalent to the baseline. That is where sequential testing correction helps control this risk by adjusting the level of confidence required before declaring something as significant.<\/p>\n","protected":false},"author":737,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-2260","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"acf":[],"yoast_head":"\nSequential Testing Correction | VWO<\/title>\n<meta name=\"description\" content=\"Sequential testing correction prevents false conclusions by adjusting decision probabilities, mitigating the risks of false winners in A\/B testing.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Sequential Testing Correction | VWO\" \/>\n<meta property=\"og:description\" content=\"Sequential testing correction prevents false conclusions by adjusting decision probabilities, mitigating the risks of false winners in A\/B testing.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/\" \/>\n<meta property=\"og:site_name\" content=\"VWO Glossary\" \/>\n<meta property=\"article:published_time\" content=\"2024-03-14T09:34:57+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-07-08T13:08:33+00:00\" \/>\n<meta name=\"author\" content=\"Aastha Trehan\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/\"},\"author\":{\"name\":\"Aastha Trehan\",\"@id\":\"https:\/\/vwo.com\/glossary\/#\/schema\/person\/95c6d9ae0e137e8e23a76689292da18b\"},\"headline\":\"Sequential Testing Correction\",\"datePublished\":\"2024-03-14T09:34:57+00:00\",\"dateModified\":\"2025-07-08T13:08:33+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/\"},\"wordCount\":877,\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/\",\"url\":\"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/\",\"name\":\"Sequential Testing Correction | VWO\",\"isPartOf\":{\"@id\":\"https:\/\/vwo.com\/glossary\/#website\"},\"datePublished\":\"2024-03-14T09:34:57+00:00\",\"dateModified\":\"2025-07-08T13:08:33+00:00\",\"author\":{\"@id\":\"https:\/\/vwo.com\/glossary\/#\/schema\/person\/95c6d9ae0e137e8e23a76689292da18b\"},\"description\":\"Sequential testing correction prevents false conclusions by adjusting decision probabilities, mitigating the risks of false winners in A\/B testing.\",\"breadcrumb\":{\"@id\":\"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/vwo.com\/glossary\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Sequential Testing Correction\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/vwo.com\/glossary\/#website\",\"url\":\"https:\/\/vwo.com\/glossary\/\",\"name\":\"VWO Glossary\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/vwo.com\/glossary\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/vwo.com\/glossary\/#\/schema\/person\/95c6d9ae0e137e8e23a76689292da18b\",\"name\":\"Aastha Trehan\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/vwo.com\/glossary\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/88aead8d5e767617fcf4e13e093c9652c9028cc733abc85e20132b3ca1f5e005?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/88aead8d5e767617fcf4e13e093c9652c9028cc733abc85e20132b3ca1f5e005?s=96&d=mm&r=g\",\"caption\":\"Aastha Trehan\"},\"description\":\"Hi there, I\u2019m Aastha Trehan, Associate Manager of Content at VWO. I write about CRO, experimentation, user behavior research, personalization, and AI-driven optimization. With over 3 years of experience in marketing, I focus on creating engaging content that resonates with readers and drives meaningful impact. When I\u2019m not working, you\u2019ll find me playing badminton or pickleball, or binge-watching movies.\",\"sameAs\":[\"https:\/\/www.linkedin.com\/in\/aastha-trehan-7b3190283\/\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Sequential Testing Correction | VWO","description":"Sequential testing correction prevents false conclusions by adjusting decision probabilities, mitigating the risks of false winners in A\/B testing.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/","og_locale":"en_US","og_type":"article","og_title":"Sequential Testing Correction | VWO","og_description":"Sequential testing correction prevents false conclusions by adjusting decision probabilities, mitigating the risks of false winners in A\/B testing.","og_url":"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/","og_site_name":"VWO Glossary","article_published_time":"2024-03-14T09:34:57+00:00","article_modified_time":"2025-07-08T13:08:33+00:00","author":"Aastha Trehan","twitter_card":"summary_large_image","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/#article","isPartOf":{"@id":"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/"},"author":{"name":"Aastha Trehan","@id":"https:\/\/vwo.com\/glossary\/#\/schema\/person\/95c6d9ae0e137e8e23a76689292da18b"},"headline":"Sequential Testing Correction","datePublished":"2024-03-14T09:34:57+00:00","dateModified":"2025-07-08T13:08:33+00:00","mainEntityOfPage":{"@id":"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/"},"wordCount":877,"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/","url":"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/","name":"Sequential Testing Correction | VWO","isPartOf":{"@id":"https:\/\/vwo.com\/glossary\/#website"},"datePublished":"2024-03-14T09:34:57+00:00","dateModified":"2025-07-08T13:08:33+00:00","author":{"@id":"https:\/\/vwo.com\/glossary\/#\/schema\/person\/95c6d9ae0e137e8e23a76689292da18b"},"description":"Sequential testing correction prevents false conclusions by adjusting decision probabilities, mitigating the risks of false winners in A\/B testing.","breadcrumb":{"@id":"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/vwo.com\/glossary\/sequential-testing-correction\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/vwo.com\/glossary\/sequential-testing-correction\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/vwo.com\/glossary\/"},{"@type":"ListItem","position":2,"name":"Sequential Testing Correction"}]},{"@type":"WebSite","@id":"https:\/\/vwo.com\/glossary\/#website","url":"https:\/\/vwo.com\/glossary\/","name":"VWO Glossary","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/vwo.com\/glossary\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/vwo.com\/glossary\/#\/schema\/person\/95c6d9ae0e137e8e23a76689292da18b","name":"Aastha Trehan","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/vwo.com\/glossary\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/88aead8d5e767617fcf4e13e093c9652c9028cc733abc85e20132b3ca1f5e005?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/88aead8d5e767617fcf4e13e093c9652c9028cc733abc85e20132b3ca1f5e005?s=96&d=mm&r=g","caption":"Aastha Trehan"},"description":"Hi there, I\u2019m Aastha Trehan, Associate Manager of Content at VWO. I write about CRO, experimentation, user behavior research, personalization, and AI-driven optimization. With over 3 years of experience in marketing, I focus on creating engaging content that resonates with readers and drives meaningful impact. When I\u2019m not working, you\u2019ll find me playing badminton or pickleball, or binge-watching movies.","sameAs":["https:\/\/www.linkedin.com\/in\/aastha-trehan-7b3190283\/"]}]}},"_links":{"self":[{"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/posts\/2260","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/users\/737"}],"replies":[{"embeddable":true,"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/comments?post=2260"}],"version-history":[{"count":3,"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/posts\/2260\/revisions"}],"predecessor-version":[{"id":4969,"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/posts\/2260\/revisions\/4969"}],"wp:attachment":[{"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/media?parent=2260"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/categories?post=2260"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/vwo.com\/glossary\/wp-json\/wp\/v2\/tags?post=2260"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}