VWO Logo
Dashboard

VWO SmartStats for Smarter Business Decisions

VWO’s Bayesian-powered statistics engine is designed to do the heavy lifting when it comes to calculations and accuracy for you and gives you all the ingredients you need to make the right business decisions.

Know more
Complies with:
VWO GDPR Ready Badge
VWO CCPA Ready Badge
VWO G2Crowd Leader Spring Badge
Trustradius badge for VWO

We tweet useful stuff daily

Related content:

Twyman’s Law

2 Min Read

Hi 👋  I am Paras Chopra, founder & chairman of VWO. Every fortnight, on this blog and on our email list, I’ll be posting a new idea or a story on experimentation and growth.

Twyman’s law states that any data or figure that looks interesting or different is usually wrong.

Sounds unbelievable, isn’t it? 

But, it’s true. I saw this in action recently and wanted to share that story with you.

In June, we ran a test on our homepage and while I was looking at conversion rate by segments, I noticed that users from Windows had a 400% higher signup rate for VWO free trial as compared to users using Mac OS X.  

Now, that’s baffling and our team spent a good deal of time trying to understand why that was happening. Someone in marketing hypothesized that perhaps Mac OS X users have a better design aesthetic and our homepage wasn’t appealing to them. Was it true?

When we dug into data, we realized that our recently installed automated QA service creates signups on the homepage every hour or so (to ensure the form doesn’t stop working) and guess what, that automated service used Windows

After removing such QA signups from data, Mac OS X and Windows conversion rate became comparable.

This is a perfect example of Twyman’s law. Remember, if the data is too good to be true, it’s probably wrong. 

Many extreme results are likely to be the result of an error in instrumentation (e.g., logging), loss of data (or duplication of data), or a computational error. 

Hope this mental model was new to you (it certainly was to me!). 

PS: If you also have a story related to exaggerated data to share, email me at paras@vwo.com. I read and reply to all emails 🙂

PPS: If you want to learn more about Twyman’s law, Ronny from Bing’s experimentation team spoke about it in his talk.

Paras Chopra
Paras Chopra Founder and Chairman of VWO
More from VWO
Why is Mobile Optimization Important for Website Conversions?

Why is Mobile Optimization Important for Website Conversions?

Just as retailers now see VoIP phone as the standard when it comes to customer…

Read More
Sam O'Brien

Sam O'Brien

6 Min Read
Experimentation ROI: How To Think Beyond Uplift

Experimentation ROI: How To Think Beyond Uplift

Ian Freed was devastated. He was at the helm of a product experiment that cost…

Read More
Mani Makkar

Mani Makkar

8 Min Read
Budgeting for CRO: Why You Need To Set Aside a Budget and How To Create One

Budgeting for CRO: Why You Need To Set Aside a Budget and How To Create One

Conversion Rate Optimization (CRO) is still under-prioritized at many companies and is often the last,…

Read More
Niti Sharma

Niti Sharma

15+ Min Read

Scale your A/B testing and experimentation with VWO.

Start Free Trial Request Demo
Shanaz Khan from VWO

Hi, I am Shanaz from the VWO Research Desk.

Join our community of 10,000+ Marketing, Product & UX Folks today & never miss the latest from the world of experience optimization.

A value for this field is required.

Thank you!

Check your inbox for the confirmation mail