VWO Logo
Dashboard
Follow us and stay on top of everything CRO
Related content:

Twyman’s Law

2 Min Read

Hi 👋  I am Paras Chopra, founder & chairman of VWO. Every fortnight, on this blog and on our email list, I’ll be posting a new idea or a story on experimentation and growth.

Twyman’s law states that any data or figure that looks interesting or different is usually wrong.

Sounds unbelievable, isn’t it? 

But, it’s true. I saw this in action recently and wanted to share that story with you.

In June, we ran a test on our homepage and while I was looking at conversion rate by segments, I noticed that users from Windows had a 400% higher signup rate for VWO free trial as compared to users using Mac OS X.  

Now, that’s baffling and our team spent a good deal of time trying to understand why that was happening. Someone in marketing hypothesized that perhaps Mac OS X users have a better design aesthetic and our homepage wasn’t appealing to them. Was it true?

When we dug into data, we realized that our recently installed automated QA service creates signups on the homepage every hour or so (to ensure the form doesn’t stop working) and guess what, that automated service used Windows

After removing such QA signups from data, Mac OS X and Windows conversion rate became comparable.

This is a perfect example of Twyman’s law. Remember, if the data is too good to be true, it’s probably wrong. 

Many extreme results are likely to be the result of an error in instrumentation (e.g., logging), loss of data (or duplication of data), or a computational error. 

Hope this mental model was new to you (it certainly was to me!). 

PS: If you also have a story related to exaggerated data to share, email me at paras@vwo.com. I read and reply to all emails 🙂

PPS: If you want to learn more about Twyman’s law, Ronny from Bing’s experimentation team spoke about it in his talk.

Paras Chopra
Paras Chopra I started Wingify in early 2009 to enable businesses to design and deploy great customer experiences for their websites and apps. I have a background in machine learning and am a gold medalist from Delhi College of Engineering. I have been featured twice in the Forbes 30 under 30 list - India and Asia. I'm an entrepreneur by profession and my curiosity is wide-ranging. Follow me at @paraschopra on Twitter. You can email me at paras@wingify.com
More from VWO
VWO Shines in the G2 Summer 2022 Reports With 20 Badges in Five Categories

VWO Shines in the G2 Summer 2022 Reports With 20 Badges in Five Categories

VWO, the world-leading experimentation platform emerged as the Leader in five categories, including Mobile App…

Read More
Mareen Cherian

Mareen Cherian

5 Min Read
5 Marketing Strategies To Grow Your Business During a Recession

5 Marketing Strategies To Grow Your Business During a Recession

The U.S GDP expanded at 2.6% in the Q3 of 2022, contrary to the growth…

Read More
Pratyusha Guha

Pratyusha Guha

10 Min Read
How to Really Win at Experimentation (Even When You Don’t) – Approach and Inspirations

How to Really Win at Experimentation (Even When You Don’t) – Approach and Inspirations

To improve conversions and keep up with the ever-changing customer behavior, enterprises are increasingly embracing…

Read More
Pratyusha Guha

Pratyusha Guha

12 Min Read

Scale your A/B testing and experimentation with VWO.

Start Free Trial Request Demo
Invalid Email

DOWNLOAD A/B TESTING FREE E-BOOK

FREE A/B TESTING KIT

All you Need to Know About Conversion Optimization Get the Guide