{"id":2123,"date":"2023-11-30T07:11:41","date_gmt":"2023-11-30T07:11:41","guid":{"rendered":"https:\/\/vwo.com\/glossary\/?p=2123"},"modified":"2025-07-08T12:54:41","modified_gmt":"2025-07-08T12:54:41","slug":"sample-ratio-mismatch","status":"publish","type":"post","link":"https:\/\/vwo.com\/glossary\/sample-ratio-mismatch\/","title":{"rendered":"Sample Ratio Mismatch"},"content":{"rendered":"\n

What is Sample Ratio Mismatch?<\/h2>\n\n\n\n

Sample Ratio Mismatch (SRM) in the context of an A\/B test<\/a> refers to an imbalance in the distribution of users between the control and variation groups. It happens when the intended randomization fails, leading to unequal sample sizes<\/a> in a test.\u00a0<\/p>\n\n\n\n

For example, you assign 50% of users to the control<\/a> group and 50% to the variation group for an A\/B test. However due to some issues, the actual distribution results in allocating 45% of users in the control group and 55% in the treatment group. This is a case of SRM, affecting the accuracy and reliability of your test results. <\/p>\n\n\n\n

Another scenario is when the configured allocation is, say, 60:40 in the A\/B test, but the observed allocation turns out to be 70:30. Any deviation from the planned distribution is considered an SRM issue.<\/p>\n\n\n\n

\n
\"Sample<\/figure>\n
Image source: Doordash<\/a>
<\/figcaption><\/figure>\n\n\n\n

What causes SRM issues?<\/h2>\n\n\n\n

There may be several reasons why SRM creeps into your A\/B test. Let\u2019s look at some of the classic reasons why this happens below:<\/p>\n\n\n\n

User behavior<\/h3>\n\n\n\n

If users delete or block cookies, it can disrupt the tracking and randomization process, leading to a sample ratio mismatch. This is because regular clearing of cookies may lead to the counting of such users as new users leading to their overrepresentation in one group. <\/p>\n\n\n\n

Technical bugs <\/h3>\n\n\n\n

Technical issues can also cause an SRM. Consider a test with JavaScript code that\u2019s making one variation crash. Due to this, some visitors sent to the crashing variant may not be recorded properly, causing SRM. <\/p>\n\n\n\n

Geographic or time differences<\/h3>\n\n\n\n

Geographic or time differences can influence user behaviors, affecting the distribution of users across groups in the A\/B test. So, for example, consider an online retail website with a global user base. If your test does not account for time zone differences, it may unwittingly include a significant number of users from a specific region in one group during certain hours. This could result in an SRM in the segment of users coming from that particular location.<\/p>\n\n\n\n

Browser or device biases <\/h3>\n\n\n\n

When specific browsers or devices are overrepresented due to biases in the randomization process, the integrity of the test can be compromised. For example, suppose you run an A\/B test on your SaaS website<\/a> for mobile but its slow loading speed led to a decreased sample allocation to the mobile variation. Without careful randomization, one group ends up with a higher proportion of users due to device or browser issues, skewing the test results. <\/p>\n\n\n\n

Dogfooding <\/h3>\n\n\n\n

Employees, being internal users, are exposed to the latest features or tests by default. As they interact with the product more frequently than external users, their inclusion in the treatment group significantly skews the metrics. This inadvertent inclusion of one’s own company\u2019s employees in a test, also known as dogfooding, can distort test results and lead to an overestimation of the impact of a test. <\/p>\n\n\n\n

When is SRM a problem and when is it not? <\/h2>\n\n\n\n

Put simply, SRM arises when one version of a test receives a significantly different number of visitors than originally expected. A classic A\/B test has a 50\/50 traffic split between two variations. <\/p>\n\n\n\n

But you see that toward the end of the test, the control gets 5,000 visitors, and the variation gets 4982 visitors. Would you call this a massive problem? Not really. <\/p>\n\n\n\n

In the final stage of an A\/B test, a slight deviation in traffic allocation can happen due to the inherent randonment in allocation. So, if you see, that the majority of traffic is rightly allocated (calculated confidence being 95%-99%), you need not worry about a slight difference in sample ratios. <\/p>\n\n\n\n

But SRM becomes a notable issue when the difference in traffic is substantial, such as 5,000 visitors directed to one version and 2100 to the other. <\/p>\n\n\n\n

That\u2019s why staying alert, and keeping an eye on visitor count is so important if you want to obtain accurate test results. <\/p>\n\n\n\n

Want to watch how you can split traffic for your A\/B test on VWO<\/a>? Here is a video for you:<\/p>\n\n\n\n

\n