CUPED

Controlled experiment using pre-experiment data (CUPED) is a variance reduction technique used in A/B testing. 

Developed by Microsoft’s data science team in the early 2010s, CUPED was created to address the need for more efficient A/B testing on platforms like Bing and Microsoft Office. Since its inception, this technique has become regular within the A/B testing and optimization communities due to its ability to reduce variance.

How does CUPED work?

Let’s try to understand how CUPED works through an example. Suppose you run an online store and want to test a new checkout process. You set up an A/B test where half of your visitors see the new checkout process (Group B) and the other half see the current one (Group A). The goal is to determine if the new checkout process leads to more completed purchases.

Before starting the test, you already have extensive data about your visitors’ behavior. For instance, you know how many purchases each visitor made in the month prior to the test. Here’s where CUPED comes into play. For each visitor in both Group A and Group B, CUPED gathers data on their purchase behavior from the previous month. As the test runs, it counts the purchases each group makes during the test period. However, instead of just comparing the raw numbers, CUPED adjusts these figures based on an increase or decrease in the numbers compared to the last month in the control group and the variation group.

Without CUPED, if Group A (current checkout) averages 10 purchases and Group B (new checkout) averages 12 purchases after the test, you might conclude that the new checkout is slightly better. But with CUPED, you adjust these numbers using the pre-experiment data. Perhaps Group A’s visitors made an average of 4 purchases, and Group B’s visitors made an average of 2 purchases before the test. After adjusting for this pre-experiment data, you might find that Group B’s improvement is even more significant.

Thus, CUPED helps you make your A/B tests more accurate and reliable by factoring in what you already know about your visitors. 

Benefits of CUPED

Here are the benefits of using CUPED to make your A/B tests more accurate and reliable:

• CUPED leverages pre-experiment data to control for natural variations in your visitors’ behavior. This means that if there’s a genuine difference between your test groups, CUPED makes it easier to spot. For instance, if your new checkout process is indeed better, CUPED will help you see that improvement more clearly.
• Reaching statistical significance requires a large number of visitors. However, with CUPED, you can achieve meaningful conclusions with fewer visitors because it reduces the “noise” from natural variations. This makes your tests more efficient and less resource-intensive.

Limitations of using CUPED

While CUPED offers significant benefits, it’s important to understand its limitations. Here are two key points to keep in mind:

• CUPED relies on pre-experiment data to reduce variance and improve the accuracy of your test results. This means it can only be used with visitors who have been to your site before. If you have a lot of new visitors, CUPED won’t be effective because there’s no past data to leverage.

• It is not effective for binary metrics, like conversion rates, because it relies on continuous data (such as the number of purchases) to adjust for pre-experiment differences. This makes it less suitable for scenarios where you’re measuring simple yes/no outcomes.

Conclusion

In conclusion, CUPED is a powerful technique that leverages pre-experiment data to enhance the accuracy and efficiency of A/B testing. It helps control variance and enables you to draw meaningful conclusions with fewer participants. However, keep in mind that CUPED is only effective with past visitors and not be suitable for binary metrics.