{"id":52531,"date":"2020-07-16T15:08:51","date_gmt":"2020-07-16T09:38:51","guid":{"rendered":"https:\/\/vwo.com\/blog\/?p=52531"},"modified":"2025-11-13T16:13:41","modified_gmt":"2025-11-13T10:43:41","slug":"twyman-law","status":"publish","type":"post","link":"https:\/\/vwo.com\/blog\/twyman-law\/","title":{"rendered":"Twyman\u2019s Law"},"content":{"rendered":"\n

Hi ?  I am Paras Chopra<\/a>, founder & chairman of VWO<\/a>. Every fortnight, on this blog and on our email list, I’ll be posting a new idea or a story <\/a>on experimentation and growth.<\/em><\/p>\n\n\n\n

Twyman\u2019s law states that any data or figure that looks interesting or different is usually wrong.<\/strong><\/p>\n\n\n\n

Sounds unbelievable, isn\u2019t it? <\/p>\n\n\n\n

But, it\u2019s true. I saw this in action recently and wanted to share that story with you.<\/p>\n\n\n\n

In June, we ran a test on our homepage and while I was looking at conversion rate by segments, I noticed that users from Windows had a 400% higher signup rate<\/strong> for VWO free trial as compared to users using Mac OS X.  <\/p>\n\n\n\n

Download Free: A\/B Testing Guide<\/span><\/a><\/em><\/h3>\n\n\n\n

Now, that’s baffling and our team spent a good deal of time trying to understand why that was happening. Someone in marketing hypothesized that perhaps Mac OS X users have a better design aesthetic and our homepage wasn\u2019t appealing to them. Was it true?<\/strong><\/p>\n\n\n\n

When we dug into data, we realized that our recently installed automated QA service creates signups on the homepage every hour or so (to ensure the form doesn\u2019t stop working) and guess what, that automated service used Windows<\/strong>. <\/p>\n\n\n\n

After removing such QA signups from data, Mac OS X and Windows conversion rate became comparable.<\/p>\n\n\n\n

This is a perfect example of Twyman’s law. Remember, if the data is too good to be true, it’s probably wrong. <\/p>\n\n\n\n

Many extreme results are likely to be the result of an error<\/strong> in instrumentation (e.g., logging), loss of data (or duplication of data), or a computational error. <\/p>\n\n\n\n

Hope this mental model was new to you (it certainly was to me!). <\/p>\n\n\n\n

PS: If you also have a story related to exaggerated data to share, email me at paras@vwo.com. I read and reply to all emails \ud83d\ude42<\/p>\n\n\n\n

PPS: If you want to learn more about Twyman\u2019s law, Ronny from Bing\u2019s experimentation team spoke about it in his talk<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

Twyman\u2019s law states that any data or figure that looks interesting or different is usually wrong. Sounds unbelievable, isn\u2019t it? <\/p>\n","protected":false},"author":14,"featured_media":55902,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"post_read_time":2,"footnotes":""},"categories":[10185],"tags":[],"feature":[],"industry-type":[],"product":[],"role":[],"region":[],"class_list":["post-52531","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-letters-from-the-founder"],"acf":[],"yoast_head":"\nTwyman's Law | VWO<\/title>\n<meta name=\"description\" content=\"Remember, if the data is too good to be true, it\u2019s probably wrong. Twyman\u2019s law states that any data or figure that looks interesting or different is usually wrong. Read more.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/vwo.com\/blog\/twyman-law\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Twyman's Law | VWO\" \/>\n<meta property=\"og:description\" content=\"Remember, if the data is too good to be true, it\u2019s probably wrong. Twyman\u2019s law states that any data or figure that looks interesting or different is usually wrong. Read more.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/vwo.com\/blog\/twyman-law\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/vwoofficial\/\" \/>\n<meta property=\"article:published_time\" content=\"2020-07-16T09:38:51+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-11-13T10:43:41+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/07\/OG-image_Twymans-Law.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Paras Chopra\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/07\/OG-image_Twymans-Law.jpg\" \/>\n<meta name=\"twitter:creator\" content=\"@paraschopra\" \/>\n<meta name=\"twitter:site\" content=\"@VWO\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Paras Chopra\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/vwo.com\/blog\/twyman-law\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/vwo.com\/blog\/twyman-law\/\"},\"author\":{\"name\":\"Paras Chopra\",\"@id\":\"https:\/\/vwo.com\/blog\/#\/schema\/person\/6a4fd0d62df48bca6a31f484f7c4e5b6\"},\"headline\":\"Twyman\u2019s Law\",\"datePublished\":\"2020-07-16T09:38:51+00:00\",\"dateModified\":\"2025-11-13T10:43:41+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/vwo.com\/blog\/twyman-law\/\"},\"wordCount\":347,\"publisher\":{\"@id\":\"https:\/\/vwo.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/vwo.com\/blog\/twyman-law\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/07\/Feature-image_Twymans-Law.png\",\"articleSection\":[\"Letters From the Founder\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/vwo.com\/blog\/twyman-law\/\",\"url\":\"https:\/\/vwo.com\/blog\/twyman-law\/\",\"name\":\"Twyman's Law | VWO\",\"isPartOf\":{\"@id\":\"https:\/\/vwo.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/vwo.com\/blog\/twyman-law\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/vwo.com\/blog\/twyman-law\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/07\/Feature-image_Twymans-Law.png\",\"datePublished\":\"2020-07-16T09:38:51+00:00\",\"dateModified\":\"2025-11-13T10:43:41+00:00\",\"description\":\"Remember, if the data is too good to be true, it\u2019s probably wrong. Twyman\u2019s law states that any data or figure that looks interesting or different is usually wrong. Read more.\",\"breadcrumb\":{\"@id\":\"https:\/\/vwo.com\/blog\/twyman-law\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/vwo.com\/blog\/twyman-law\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/vwo.com\/blog\/twyman-law\/#primaryimage\",\"url\":\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/07\/Feature-image_Twymans-Law.png\",\"contentUrl\":\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/07\/Feature-image_Twymans-Law.png\",\"width\":1200,\"height\":700,\"caption\":\"Twyman\u2019s Law\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/vwo.com\/blog\/twyman-law\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/vwo.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Letters From the Founder\",\"item\":\"https:\/\/vwo.com\/blog\/letters-from-the-founder\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Twyman\u2019s Law\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/vwo.com\/blog\/#website\",\"url\":\"https:\/\/vwo.com\/blog\/\",\"name\":\"Blog\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/vwo.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/vwo.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/vwo.com\/blog\/#organization\",\"name\":\"VWO\",\"url\":\"https:\/\/vwo.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/vwo.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2018\/09\/VWOLogo.png\",\"contentUrl\":\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2018\/09\/VWOLogo.png\",\"width\":780,\"height\":492,\"caption\":\"VWO\"},\"image\":{\"@id\":\"https:\/\/vwo.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/vwoofficial\/\",\"https:\/\/x.com\/VWO\",\"https:\/\/www.instagram.com\/vwoofficial\/\",\"https:\/\/www.linkedin.com\/company\/vwo\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/vwo.com\/blog\/#\/schema\/person\/6a4fd0d62df48bca6a31f484f7c4e5b6\",\"name\":\"Paras Chopra\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/vwo.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2019\/04\/paras-150x150.jpg\",\"contentUrl\":\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2019\/04\/paras-150x150.jpg\",\"caption\":\"Paras Chopra\"},\"description\":\"I started Wingify in early 2009 to enable businesses to design and deploy great customer experiences for their websites and apps. I have a background in machine learning and am a gold medalist from Delhi College of Engineering. I have been featured twice in the Forbes 30 under 30 list - India and Asia. I'm an entrepreneur by profession and my curiosity is wide-ranging. Follow me at @paraschopra on Twitter. You can email me at paras@wingify.com\",\"sameAs\":[\"https:\/\/invertedpassion.com\/\",\"https:\/\/in.linkedin.com\/in\/paraschopra\",\"https:\/\/x.com\/paraschopra\"],\"url\":\"https:\/\/vwo.com\/blog\/author\/paraschopra\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Twyman's Law | VWO","description":"Remember, if the data is too good to be true, it\u2019s probably wrong. Twyman\u2019s law states that any data or figure that looks interesting or different is usually wrong. Read more.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/vwo.com\/blog\/twyman-law\/","og_locale":"en_US","og_type":"article","og_title":"Twyman's Law | VWO","og_description":"Remember, if the data is too good to be true, it\u2019s probably wrong. Twyman\u2019s law states that any data or figure that looks interesting or different is usually wrong. Read more.","og_url":"https:\/\/vwo.com\/blog\/twyman-law\/","og_site_name":"Blog","article_publisher":"https:\/\/www.facebook.com\/vwoofficial\/","article_published_time":"2020-07-16T09:38:51+00:00","article_modified_time":"2025-11-13T10:43:41+00:00","og_image":[{"width":1200,"height":630,"url":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/07\/OG-image_Twymans-Law.png","type":"image\/png"}],"author":"Paras Chopra","twitter_card":"summary_large_image","twitter_image":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/07\/OG-image_Twymans-Law.jpg","twitter_creator":"@paraschopra","twitter_site":"@VWO","twitter_misc":{"Written by":"Paras Chopra","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/vwo.com\/blog\/twyman-law\/#article","isPartOf":{"@id":"https:\/\/vwo.com\/blog\/twyman-law\/"},"author":{"name":"Paras Chopra","@id":"https:\/\/vwo.com\/blog\/#\/schema\/person\/6a4fd0d62df48bca6a31f484f7c4e5b6"},"headline":"Twyman\u2019s Law","datePublished":"2020-07-16T09:38:51+00:00","dateModified":"2025-11-13T10:43:41+00:00","mainEntityOfPage":{"@id":"https:\/\/vwo.com\/blog\/twyman-law\/"},"wordCount":347,"publisher":{"@id":"https:\/\/vwo.com\/blog\/#organization"},"image":{"@id":"https:\/\/vwo.com\/blog\/twyman-law\/#primaryimage"},"thumbnailUrl":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/07\/Feature-image_Twymans-Law.png","articleSection":["Letters From the Founder"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/vwo.com\/blog\/twyman-law\/","url":"https:\/\/vwo.com\/blog\/twyman-law\/","name":"Twyman's Law | VWO","isPartOf":{"@id":"https:\/\/vwo.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/vwo.com\/blog\/twyman-law\/#primaryimage"},"image":{"@id":"https:\/\/vwo.com\/blog\/twyman-law\/#primaryimage"},"thumbnailUrl":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/07\/Feature-image_Twymans-Law.png","datePublished":"2020-07-16T09:38:51+00:00","dateModified":"2025-11-13T10:43:41+00:00","description":"Remember, if the data is too good to be true, it\u2019s probably wrong. Twyman\u2019s law states that any data or figure that looks interesting or different is usually wrong. Read more.","breadcrumb":{"@id":"https:\/\/vwo.com\/blog\/twyman-law\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/vwo.com\/blog\/twyman-law\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/vwo.com\/blog\/twyman-law\/#primaryimage","url":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/07\/Feature-image_Twymans-Law.png","contentUrl":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/07\/Feature-image_Twymans-Law.png","width":1200,"height":700,"caption":"Twyman\u2019s Law"},{"@type":"BreadcrumbList","@id":"https:\/\/vwo.com\/blog\/twyman-law\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/vwo.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Letters From the Founder","item":"https:\/\/vwo.com\/blog\/letters-from-the-founder\/"},{"@type":"ListItem","position":3,"name":"Twyman\u2019s Law"}]},{"@type":"WebSite","@id":"https:\/\/vwo.com\/blog\/#website","url":"https:\/\/vwo.com\/blog\/","name":"Blog","description":"","publisher":{"@id":"https:\/\/vwo.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/vwo.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/vwo.com\/blog\/#organization","name":"VWO","url":"https:\/\/vwo.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/vwo.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2018\/09\/VWOLogo.png","contentUrl":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2018\/09\/VWOLogo.png","width":780,"height":492,"caption":"VWO"},"image":{"@id":"https:\/\/vwo.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/vwoofficial\/","https:\/\/x.com\/VWO","https:\/\/www.instagram.com\/vwoofficial\/","https:\/\/www.linkedin.com\/company\/vwo"]},{"@type":"Person","@id":"https:\/\/vwo.com\/blog\/#\/schema\/person\/6a4fd0d62df48bca6a31f484f7c4e5b6","name":"Paras Chopra","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/vwo.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2019\/04\/paras-150x150.jpg","contentUrl":"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2019\/04\/paras-150x150.jpg","caption":"Paras Chopra"},"description":"I started Wingify in early 2009 to enable businesses to design and deploy great customer experiences for their websites and apps. I have a background in machine learning and am a gold medalist from Delhi College of Engineering. I have been featured twice in the Forbes 30 under 30 list - India and Asia. I'm an entrepreneur by profession and my curiosity is wide-ranging. Follow me at @paraschopra on Twitter. You can email me at paras@wingify.com","sameAs":["https:\/\/invertedpassion.com\/","https:\/\/in.linkedin.com\/in\/paraschopra","https:\/\/x.com\/paraschopra"],"url":"https:\/\/vwo.com\/blog\/author\/paraschopra\/"}]}},"_links":{"self":[{"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/posts\/52531","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/users\/14"}],"replies":[{"embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/comments?post=52531"}],"version-history":[{"count":18,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/posts\/52531\/revisions"}],"predecessor-version":[{"id":102554,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/posts\/52531\/revisions\/102554"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/media\/55902"}],"wp:attachment":[{"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/media?parent=52531"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/categories?post=52531"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/tags?post=52531"},{"taxonomy":"feature","embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/feature?post=52531"},{"taxonomy":"industry-type","embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/industry-type?post=52531"},{"taxonomy":"product","embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/product?post=52531"},{"taxonomy":"role","embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/role?post=52531"},{"taxonomy":"region","embeddable":true,"href":"https:\/\/vwo.com\/blog\/wp-json\/wp\/v2\/region?post=52531"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}