VWO Testing
A/B Testing

A/B Testing Solutions to Maximize Your Website Conversions

Create and A/B test different versions of your website and its elements to continuously discover the best-performing versions that improve your conversions.

Know More
VWO GDPR Ready Badge
VWO CCPA Ready Badge
VWO G2Crowd Leader Spring Badge
VWO BS 10012 Certification Badge
SOC 2 logo
Follow us and stay on top of everything CRO
Related content:

How To Run 25,000 A/B Tests in 2024

4 Min Read

Hi 👋 I am Paras Chopra, founder & chairman of VWO. This is the 6th installment of my fortnightly series outlining a new idea or a story on experimentation and growth. Hope you enjoy reading it!

There’s a company that has grown twice as fast as S&P 500. Beating the market consistently for more than 10 years proves that what they’re doing isn’t a fluke.

growth of booking.com versus S&P 500

This company – whose stock price is shown as the blue curve in the graph above – must be having a secret sauce for growth. 

Download Free: A/B Testing Guide

What is it?

Well, the company in question is Booking.com, and they are known for their culture of experimentation. They collectively run more than 25,000 A/B tests annually – which means they’re launching ~70 tests every single day. Sounds crazy, but according to their leadership, this A/B testing craziness is one of the critical ingredients in their growth.

To see how this culture of experimentation has played a role in their growth, look at their hotel listings page.

hotel listings page on boooking.com
Image Source: Booking.com

Everything highlighted on the listing page is there either because an A/B test found it made an impact on their business or is currently getting A/B tested. More importantly, everything that you don’t find on the page is not there because an A/B test proved that it actually hurt their business or made no difference.

To learn more about how Booking.com scaled its testing program to 25,000 tests a year, you can check these two resources:

So, how can you too run 25,000 tests this year?

Well, I understand that not every website has traffic levels of Booking.com. But if not 25,000 A/B tests a year, a site with one-hundredth of Booking.com’s traffic should at least be running one-hundredth the tests – say 250 tests a year, which is almost 20 tests a month. 

Do you run 20 A/B tests a month? What about 10 per month?

Unless you’re an agency or an outlier company, my guess is that the answer would be no.

I see many businesses running occasional A/B tests, and if they have a program, they’d be perhaps launching 2-3 A/B tests a month. That’s about it. Finding exceptional cultures where experimentation is taken seriously is a rarity. But why?

I think the answer boils down to whether there’s a commitment from the leadership. Experimentation is hard. Coming up with ideas, setting up tests, investing in infrastructure, getting variants designed and coded, performing quality checks, and interpreting results take real effort. Combine this with the emotional cost of most experiments not working out. No wonder so many organizations just give up on it too soon.

Download Free: A/B Testing Guide

But giving up on experimentation is giving up on a high-leverage activity that can trigger business growth. This is because an increased conversion rate is only the most direct benefit of experimentation. A full commitment to experimentation transforms the entire culture:

  • People in the org start trusting data more. 
  • A higher percentage of decisions start getting questioned on whether they’re someone’s pet projects or actual business-needle moving. 
  • You end up having fewer failed launches. 
  • And, the best part, the organization gets exposed to deliberate serendipity. Even if an A/B test didn’t win in the traditional sense, most A/B tests end up increasing the amount of insight the org has about customers and markets.

Leaders at Booking.com understand how easy it is to give up on experimentation, and that is why they’ve done things like:

  • Enabling everyone in the org to launch an A/B test without gatekeepers.
  • Having processes in place that ensure that no changes are made to any page until they’re A/B tested.
  • Make experimentation evangelizing team report into the company’s leadership.

A culture that prioritizes understanding customers better than competitors is the real benefit that experiments create. The actual conversion rate increase is simply a nice cherry on top.

So, let me ask again: what would it take for your organization to launch more A/B tests in 2023 than it did in 2022? What’s preventing your organization from adopting experimentation as a central part of your growth strategy? Send me a note at paras@vwo.com. I read and reply to all emails.

Paras Chopra
Paras Chopra I started Wingify in early 2009 to enable businesses to design and deploy great customer experiences for their websites and apps. I have a background in machine learning and am a gold medalist from Delhi College of Engineering. I have been featured twice in the Forbes 30 under 30 list - India and Asia. I'm an entrepreneur by profession and my curiosity is wide-ranging. Follow me at @paraschopra on Twitter. You can email me at paras@wingify.com
More from VWO on A/B Testing
How to Calculate A/B Testing Sample Sizes?

How to Calculate A/B Testing Sample Sizes?

(This post is a scientific explanation of the optimal sample size for your tests to…

Read More
Kees Schippers

Kees Schippers

14 Min Read
Investing in CRO or A/B Testing Software? You (Decision Maker) Must Know This

Investing in CRO or A/B Testing Software? You (Decision Maker) Must Know This

You must have come across A/B testing before. While it might not always have seemed…

Read More
Sharan Suresh

Sharan Suresh

15+ Min Read
A Practical Guide to Split Testing

A Practical Guide to Split Testing

Over the past decade, experience optimizers have crafted many strategies to effectively and efficiently increase…

Read More
Astha Khandelwal

Astha Khandelwal

9 Min Read

Scale your A/B testing and experimentation with VWO.

Start Free Trial Request Demo