VWO Logo
Dashboard
VWO Testing
A/B Testing

A/B Testing Solutions to Maximize Your Website Conversions

Create and A/B test different versions of your website and its elements to continuously discover the best-performing versions that improve your conversions.

Know More
Complies with:
VWO GDPR Ready Badge
VWO CCPA Ready Badge
VWO G2Crowd Leader Spring Badge
VWO BS 10012 Certification Badge

We tweet useful stuff daily

Related content:

No, that won’t work

3 Min Read

Hi 👋  I am Paras Chopra, founder & chairman of VWO. Here is the 7th post in my fortnightly series outlining a new idea or a story on experimentation and growth. Hope you find it valuable.

Last week I interviewed Ian Harris, a professor of surgery, for my podcast. His research has found that over half of all surgeries are ineffective.

Yes, you heard that right: the majority of surgeries don’t work (as compared to a placebo). The list of what doesn’t work includes commonly done surgeries such as those done for knee pain, back pain, hip replacement, and so on. (If you don’t believe me, I highly recommend watching this talk by the professor.)

Nl

This claim indeed sounds shocking, but recall that the most common surgical procedure in human history was bloodletting. In this practice that lasted for thousands of years and was common until the 1850s, the doctor would make a cut in the body and simply let out the blood of the patient.

Everyone thought this should work (partly because everyone did it) and many different variations of bloodletting were iterated and compared with each other. But, amazingly, a question that wasn’t asked until the 19th century was whether performing bloodletting works better than not doing it at all? Once this question was asked and a randomized control study was done, it was quickly understood that bloodletting doesn’t work (but may actually actively harm the patient).

Let me reiterate… bloodletting was performed for over 2000 years before it was discovered that it doesn’t work.

This makes you wonder – what else that we do commonly doesn’t actually work?

Actually, the sad part is that we don’t know what doesn’t work because, apart from medicine and science, we don’t perform enough randomized control trials (which are nothing but A/B tests in the marketing parlance). Even in the field of surgery, a systematic review of ~10,000 surgical procedures discovered that an A/B test was conducted for only 1% of such procedures. The remaining 99% of procedures were simply assumed to work.

In our day to day work as marketers, product managers, or leaders, we take so many decisions that feel right. Surely, that “best practice” change should work. Or this new campaign makes sense, so it should be run. But does it really work? What evidence do you have that if you hadn’t done anything, that metric wouldn’t have gone up?

Like the human body, businesses and markets are complex systems. There are thousands of factors that influence whether a particular action will have the desired impact. If the suggested action is our idea, we’re biased to believe that it works. 

However, learning from the ineffectiveness of bloodletting and many such surgeries that are still performed, the right attitude should be that best practices and genius ideas may simply not work (or, worse, may actively harm).

Unless a properly set up randomized control trial (or an A/B test) gives clear evidence of harm or benefit of a particular action, it would pay to believe that it probably doesn’t have any impact. The default orientation towards a new idea should be: no, it doesn’t work (unless you can prove that it works).

In other words, yes you should really be planning to run 25,000 A/B tests in 2021.

Do you have a story of an A/B test that debunked a widely held belief? I’d love to know about it. Send me a note at paras@vwo.com. I read and reply to all emails.

Paras Chopra
Paras Chopra I started Wingify in early 2009 to enable businesses to design and deploy great customer experiences for their websites and apps. I have a background in machine learning and am a gold medalist from Delhi College of Engineering. I have been featured twice in the Forbes 30 under 30 list - India and Asia. I'm an entrepreneur by profession and my curiosity is wide-ranging. Follow me at @paraschopra on Twitter. You can email me at paras@wingify.com
More from VWO
Future is Probabilistic

Future is Probabilistic

“All models are wrong, but some are useful”. What did the British statistician George Box…

Read More
Paras Chopra

Paras Chopra

3 Min Read
Experimentation ROI: How To Think Beyond Uplift

Experimentation ROI: How To Think Beyond Uplift

Ian Freed was devastated. He was at the helm of a product experiment that cost…

Read More
Mani Makkar

Mani Makkar

7 Min Read
How Optimizing Your Checkout Page Is Different From Optimizing Your Landing Page

How Optimizing Your Checkout Page Is Different From Optimizing Your Landing Page

Your checkout page is undeniably the “holy grail” of your website if that’s where you…

Read More
Vlad Bisceanu

Vlad Bisceanu

12 Min Read

Scale your A/B testing and experimentation with VWO.

Start Free Trial Request Demo
Invalid Email
Shanaz Khan from VWO

Hi, I am Shanaz from the VWO Research Desk.

Join our community of 10,000+ Marketing, Product & UX Folks today & never miss the latest from the world of experience optimization.

A value for this field is required.

Thank you!

Check your inbox for the confirmation mail

All you Need to Know About Conversion Optimization Get the Guide