A/B testing the covid19 vaccine

4 Min Read
Request Demo Start Free Trial

A/B testing the covid19 vaccine

4 Min Read
Request Demo Start Free Trial
Follow us and stay on top of everything CRO

Hi 👋  I am Paras Chopra, founder & chairman of VWO. I am back with the 8th post in my fortnightly series outlining a new idea or a story on experimentation and growth. Hope you find it valuable.

Vaccines for COVID-19 were first developed a year ago, and finally, they’re here. What took them so long?

The answer to this question points to the most significant A/B test conducted in recent history: the clinical trials for ensuring vaccines are safe and effective against the COVID-19 virus.

Download Free: A/B Testing Guide

Vaccine Clinical Trials 101: How do we develop and test new vaccines?

These clinical trials take so long because they’re divided into three phases, each testing a different aspect of the vaccine. This 3 stage design has taken shape over multiple decades, with thousands of scientists informing how it should be run.

Because human lives are at stake, clinical trials of vaccines are done very carefully. Be too hasty, and an unsafe vaccine can get released. But be too careful, and many more lives can be lost from the unchecked pandemic. 

This trade-off leads to the thoughtful design of clinical trials of vaccines, and hence for the conversion optimization industry, there’s a lot to learn from it.

So, let’s see the three stages in which vaccines are tested and what we can learn from them for our own A/B tests.

Phase 1: Test for side-effects

Once developed in the lab and tested on animal models, in the first phase, a vaccine is tested on a small group of humans. The primary objective of this phase is to ensure the safety of the vaccine and check if there are any side effects from it.

Translating it to the A/B testing world, if you have a large change to make (say a homepage change or a massive marketing campaign), the first thing you need to check for is not the positive impact of the change but for potential negative impact.

A variation that increases your clickthrough rate but ultimately decreases your business is no good. Hence, you should always have a set of business metrics (called guardrail metrics) which you never want to see getting reduced.

Phase 2: Test for effect size

Once phase 1 is over, researchers enroll a bigger group and divide this group into multiple groups, with each group getting a different dosage level and one group getting no dosage at all (instead of getting a placebo). You can think of this as an A/B/C/D test where the control means no change while different variations correspond to different dosage levels.

Volunteers belonging to different groups are then monitored over time to see whether the disease was cured or prevented while ensuring no major side effects.
The variation with minimum side effects and maximum benefits is declared a winner and then progressed to final phase 3 trials.

Phase 3: Large-scale confirmatory test

By phase 2, there’s clear evidence that the vaccine seems to work, and researchers have discovered an appropriate dosage for it as well. However, the work isn’t done.

Researchers want to be doubly sure that the vaccine will work when released to the masses and hence in phase 3 significantly expand the number of people included in the test. Researchers actively seek out people from various previous conditions and age groups to ensure the sample they have is a good representation of the entire population.

This larger group is divided into two groups, and the final A/B test happens with one placebo and the winner dosage from the previous phase. This is the phase that takes the most amount of time and effort.

Download Free: A/B Testing Guide

Summing up: A/B testing works!

The trade-offs relating to human lives seen for testing vaccines aren’t likely to arrive in our businesses, but we all can learn from the rigorous manner in which scientists figure out things. Just like us, they’re humans too and want so badly for vaccines and other treatments to work. But they also know that the cost of a wrong vaccine can be devastating. 

In the same spirit, I think it’s prudent to realize that even though we want our ideas to succeed so badly, we need to be careful about the possibility of such ideas making things worse for the business. A/B testing is the only way to ensure that they don’t.

If you have a story of an idea that you really wanted to work on, but it didn’t, I’d love to know about it. Send me a note at paras@vwo.com. I read and reply to all emails.

Paras Chopra
I started Wingify in early 2009 to enable businesses to design and deploy great customer experiences for their websites and apps. I have a background in machine learning and am a gold medalist from Delhi College of Engineering. I have been featured twice in the Forbes 30 under 30 list - India and Asia. I'm an entrepreneur by profession and my curiosity is wide-ranging. Follow me at @paraschopra on Twitter. You can email me at paras@wingify.com
Uncover hidden visitor insights to improve their website journey
Share
Related content
You might also love to read these
Too Much Personalization Overwhelms Users and Complicates Their Experience
10 Min Read

Too Much Personalization Overwhelms Users and Complicates Their Experience

Pratyusha Guha

Pratyusha Guha

Think Beyond Conversion Rates and Gain Richer Insights With Segmentation
12 Min Read

Think Beyond Conversion Rates and Gain Richer Insights With Segmentation

Pratyusha Guha

Pratyusha Guha

Smart Product Decisions Are Built on Iteration, Not One-Time Validation
13 Min Read

Smart Product Decisions Are Built on Iteration, Not One-Time Validation

Pratyusha Guha

Pratyusha Guha

Top 14 Product Marketing Tools for Marketers
15+ Min Read

Top 14 Product Marketing Tools for Marketers

Mareen Cherian

Mareen Cherian

Experimentation Fuels Innovation, and Failure Is Key to the Process
10 Min Read

Experimentation Fuels Innovation, and Failure Is Key to the Process

Pratyusha Guha

Pratyusha Guha

Relying on User Insights, Not Opinions, Drives Conversion Success
6 Min Read

Relying on User Insights, Not Opinions, Drives Conversion Success

Pratyusha Guha

Pratyusha Guha

8 Top Customer Journey Mapping Tools: Features, Pricing & More
14 Min Read

8 Top Customer Journey Mapping Tools: Features, Pricing & More

Ketan Pande

Ketan Pande

Introducing the New Stats Engine and enhanced VWO Reports
9 Min Read

Introducing the New Stats Engine and enhanced VWO Reports

Ishan Goel

Ishan Goel

Get new content on mail

A value for this field is required.
Thank you.

You are now subscribed to our blog.

Deliver great experiences. Grow faster, starting today.

Start Free Trial Request Demo
Shanaz Khan from VWO

Hi, I am Pratyusha from the VWO Research Desk.

Join our community of 10,000+ Marketing, Product & UX Folks today & never miss the latest from the world of experience optimization.

A value for this field is required.

Thank you!

Check your inbox for the confirmation mail