VWO Testing
A/B Testing

A/B Testing Solutions to Maximize Your Website Conversions

Create and A/B test different versions of your website and its elements to continuously discover the best-performing versions that improve your conversions.

Know More
VWO GDPR Ready Badge
VWO CCPA Ready Badge
VWO G2Crowd Leader Spring Badge
VWO BS 10012 Certification Badge
SOC 2 logo
Follow us and stay on top of everything CRO

A/B testing is not snake oil

3 Min Read

Recently on Hacker News, someone commented that A/B testing has become like snake oil, making grand promises to be a panacea for increasing conversion rates. This comment troubled me not because the commenter was wrong (he was!) but because of how certain people view A/B testing. Let me put this straight and clear:

A/B testing does NOT guarantee an increase in sales and conversions

In fact, we recently featured a guest post by Noah of Appsumo who revealed that only 1 out of 8 split tests that they run produce any actionable results. Yep, read that once again: only 1 out of 8 split tests work. Snake oil guarantees to be a cure for all diseases; A/B testing has no such guarantees.

Download Free: A/B Testing Guide

publicity of snake oil as a remedy for diseases
Image source: Neurologicablog

I think the reason why some people would view split testing suspiciously is because we release a lot of A/B testing success stories. In fact, with a user base so large (6000+ at the time of writing), we see many of our customers getting good results all the time and they if want to share those results with the world, we publish them on our blog.

So, when one week we publish a success story titled 3 dead-simple Changes increased sales by 15% and the next week we publish another success story titled How WriteWork.com increased sales by 50%, it naturally makes some people assume using VWO to do A/B testing would ensure a growth in sales. No, using VWO won’t guarantee anything. It is just an A/B testing tool (albeit, a very good one!) but the real work done is always by the craftsman, not the tool s/he uses.

Survivorship Bias: it helps to know a thing or two about it

cartoon illustrating what is the meaning of survivorship bias
Image source: Dilbert

Most success stories or case studies (be it about A/B testing, getting rich or losing weight) that you read about on the Internet suffer from something known in psychology as survivorship bias. In context of A/B testing, what it means is simple:

If you run 100 different A/B tests and only 1 of them produces good results, you are only going to publish about that one good result and not the 99 other unsuccessful results.

Obviously, 99 case studies about how A/B testing did not produce any results in unexciting and will bore our blog readers. Hence, this post is a short reminder about all those case studies that don’t produce results.

Download Free: A/B Testing Guide

Why publish A/B testing case studies at all

The reason we publish A/B testing case studies is to show the potential of what can be achieved using A/B testing. It is similar to holding an art exhibition for Picasso’s paintings which inspires people to appreciate the work and perhaps pick up a brush themselves to start painting hoping to achieve the level of success that Picasso achieved. To reiterate, just as picking up a brush and canvas does not guarantee good art, doing A/B testing does not guarantee an increase in sales.

an example of a painting
Image source: Picasso

In fact, specifically to A/B tests, replicating them almost never works. So, if you read a case study of how video increases conversions by 46% and then implement it on your site hoping for the same magical increase, it may or may not happen. What worked for them may not work for you! So, why publish case studies at all? Here’s why:

A/B testing case studies give ideas on what you can test on your website

Yep, that’s all there is to A/B testing case studies.

And, yes, I hope I convinced you that A/B testing is not snake oil!

Paras Chopra
Paras Chopra I started Wingify in early 2009 to enable businesses to design and deploy great customer experiences for their websites and apps. I have a background in machine learning and am a gold medalist from Delhi College of Engineering. I have been featured twice in the Forbes 30 under 30 list - India and Asia. I'm an entrepreneur by profession and my curiosity is wide-ranging. Follow me at @paraschopra on Twitter. You can email me at paras@wingify.com
Share
Related content
More from VWO on A/B Testing
A/B Test Proves That Women Find Bearded Men More Attractive

A/B Test Proves That Women Find Bearded Men More Attractive

Disclaimer: This is a fictional piece intended for fun reading. Yes, you heard that right.…

Read More
Mohita Nagpal

Mohita Nagpal

4 Min Read
Second Year In A Row: 9 Award Winning A/B and Multivariate Tests

Second Year In A Row: 9 Award Winning A/B and Multivariate Tests

WhichTestWon is a website that features results of A/B tests and multivariate tests wherein visitors…

Read More
Siddharth Deswal

Siddharth Deswal

2 Min Read
12 Game-Changing A/B Testing Tips for 2024

12 Game-Changing A/B Testing Tips for 2024

A new year demands fresh ideas, resolutions and plans. It’s that time of the year…

Read More
Mohita Nagpal

Mohita Nagpal

8 Min Read

Scale your A/B testing and experimentation with VWO.

Start Free Trial Request Demo
Shanaz Khan from VWO

Hi, I am Pratyusha from the VWO Research Desk.

Join our community of 10,000+ Marketing, Product & UX Folks today & never miss the latest from the world of experience optimization.

A value for this field is required.

Thank you!

Check your inbox for the confirmation mail