The CEO here is Jeff Bezos, and the failed product was the Fire Phone.<\/p>\n\n\n\n
Download Free: A\/B Testing Guide<\/span><\/a><\/em><\/h2>\n\n\n\nBefore I narrate the consequences of this expensive, failed experiment, let\u2019s take a pause and understand what ROI means. <\/p>\n\n\n\n
What is ROI?<\/strong><\/p>\n\n\n\nSimply put, ROI or Return on investment tells you whether you\u2019re getting your money\u2019s worth from your marketing initiatives. <\/p>\n\n\n\n
When the ROI goes out of control, it should set alarm bells ringing. <\/p>\n\n\n\n
Therefore, coming back to our story, I would be losing sleep over the Fire Phone if I were Jeff Bezos. Why wasn\u2019t he? The answer, and I quote him, came in a 2018 letter to Amazon\u2019s shareholders:<\/p>\n\n\n\n
\u201cAs a company grows, everything needs to scale, including the size of your failed experiments. If the size of your failures isn\u2019t growing, you\u2019re not going to be inventing at a size that can actually move the needle.\u201d<\/em><\/p>\n\n\n\n![\"Amazon\u2019s](\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/06\/Jeff-Bezos.png\")
Amazon\u2019s CEO Jeff Bezos announcing Fire smartphone at an Amazon conference<\/em>
Image source: Geekwire<\/a><\/figcaption><\/figure>\n\n\n\nGreat story, right?<\/p>\n\n\n\n
Unfortunately though, if your manager isn\u2019t Bezos, and your company doesn\u2019t have deep pockets like Amazon, you\u2019re most likely in trouble. <\/p>\n\n\n\n
Your manager might ask you questions like \u201cDid the variation win?\u201d or \u201cDid this experiment move the needle enough to put it in my town hall presentation?\u201d. If the answer to these questions is NO, the experimentation exercise<\/a> is termed futile or having no ROI. <\/p>\n\n\n\nWhy so, you ask? This is because there is so much literature around us which declares uplift (in conversion rates or revenues) as the only metric to measure the success of experimentation. Everyone expects an uplift from experiment, and then terms the uplift as either \u2018present\u2019 or \u2018absent\u2019 – there is no in-between. <\/p>\n\n\n\n
This approach reduces experimentation to a point in a checklist<\/a> which companies want to tick off. This is because they see experimentation as the goose, which will suddenly lay golden eggs (read: uplift). <\/p>\n\n\n\nThe truth is that though the industry has conditioned us to think this way, this should not be the reasoning behind any experiment.<\/p>\n\n\n
Why do we experiment?<\/h2>\n\n\n
Essentially, experiments help in achieving the following objectives: <\/p>\n\n\n
1. Making better decisions<\/strong><\/h3>\n\n\nRead that again. It\u2019s not \u2018correct\u2019 or \u2018right\u2019; it\u2019s making \u2018better\u2019 decisions. Experimentation aids data-driven decision making<\/a>, which is better than based on gut or instinct. Without an experiment, there is no way to objectively denounce an idea.<\/p>\n\n\n2. Reducing ambiguities and risk of losing business<\/strong><\/h3>\n\n\nIt\u2019s always better to test your hypotheses<\/a> than doing a full rollout of your new website or product features. This, in turn, minimizes the risk of impacting all your current business metrics – whether it\u2019s the conversion rate on your website or feature adoption rates.<\/p>\n\n\n3. Prioritizing and learning what works and what does not<\/strong><\/h3>\n\n\nBuilding an experimentation roadmap<\/a> helps you prioritize and differentiate between the must-haves and good-to-haves. Some experiments may not give you an uplift but will provide you tonnes of learnings for your next one.<\/p>\n\n\n\n![\"cartoon](\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/07\/HiPPos.jpeg\")
An illustration on Hippo: Highest Paid Person’s Opinion <\/em>
Image source: Medium<\/a>
<\/figcaption><\/figure>\n<\/div>\n\n\nThe problem arises when \u2018HiPPOs\u2019 decide which experiment to run first, and their primary expectation is a revenue uplift. The fundamental purpose of experimentation to make better decisions goes for a toss. <\/p>\n\n\n
3. Prioritizing and learning what works and what does not<\/strong><\/h3>\n\n\nImagine a situation where you, as a CRO practitioner, have been running an experimentation program for a quarter or 6 months with multiple experiments across different pages of your website. Most of these tests have yielded statistically significant winners, and you are elated. <\/p>\n\n\n\n
However, when you look at the overall conversion rate of your website, it is almost the same as a quarter before. What do you tell your manager? How do you justify the investments you made in the CRO product which you championed to buy? Does it mean your optimization initiatives are worthless? (Story of your life? Shoot me an email<\/a> or tag me on Twitter<\/a> if I nailed it.)<\/p>\n\n\n\nImagine the same situation–with a quarter spent in testing and continuous optimization–but this time your overall revenue has shot up by 3%. You are elated as this is your moment. But as soon as you enter your CEO\u2019s cabin\/zoom room, you see the head of SEO, head of performance marketing, and others already showing how they are responsible for this uplift. <\/p>\n\n\n\n
It makes you wonder and doubt how you can isolate the impact of the CRO experiments<\/a> from other variables and proclaim this as a victory for you and your team.<\/p>\n\n\n\n(Sidenote: we organized a webinar to try and unknot the mystery behind attribution challenges with experimentation<\/a>.)<\/p>\n\n\n\nIt is hard to accurately calculate the ROI of your experimentation program because of issues in forecasting, dealing with multiple variables, and running multiple experiments. CRO practitioners deal in averages, and what might seem an impact \u2018wave\u2019 for a specific segment of users might just be a ripple in the ocean when you look at global goals.<\/p>\n\n\n\n
And you\u2019re not alone. Almost 53% of CRO professionals cannot calculate ROI of experimentation<\/a>.<\/p>\n\n\n\n![\"pie](\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/06\/Experimentation-ROI_pie-chart-01.jpg\")
A pie chart showing percentages of the CRO professionals calculating ROI of experimentation<\/em>
Image source: Marketingsherpa<\/a><\/figcaption><\/figure>\n\n\n\nInstead of putting resources behind chasing that one ROI number, I would advise using those resources to increase the testing velocity. This is because the ROI on experimentation is amplified by the customer insights you gain with each experiment<\/a> you perform. That\u2019s what Bezos did with Ian Freed. <\/p>\n\n\n\nThe voice recognition software of the Fire Phone could follow commands and fetch information from the cloud. This feature was talked about and loved by the customers. It made Bezos curious, and he put Freed on a project to build a team and technology to respond to voice commands. Four months later, Echo was born. Jeff Bezos looked back at the incident and remarked: <\/p>\n\n\n\n
\u201cWhile the Fire Phone was a failure, we were able to take our learnings (as well as the developers) and accelerate our efforts building Echo and Alexa.\u201d<\/em><\/p>\n\n\n\nDownload Free: A\/B Testing Guide<\/span><\/a><\/em><\/h2>\n\n\nSo what is the impact of experimentation?<\/h2>\n\n\n![\"illustration](\"https:\/\/static.wingify.com\/gcp\/uploads\/sites\/3\/2020\/06\/experimentation-impacts-04-01-1-1024x516.jpg\")
<\/figure>\n\n\n1. Innovation\/Breakthrough<\/strong><\/h3>\n\n\nThink about the players who have disrupted industries – Netflix, Amazon, Booking, etc. The things that they have in common are a company-wide culture of experimentation and unmatched testing velocity that allows them to fail fast and move on to the next innovation. <\/p>\n\n\n\n
\u201cIf you have to kiss a lot of frogs to find a prince, find more frogs and kiss them faster and faster.\u201d ~Mike Moran<\/em><\/p>\n\n\n2. Reducing the risk of new launches<\/strong><\/h3>\n\n\n
Simply put, ROI or Return on investment tells you whether you\u2019re getting your money\u2019s worth from your marketing initiatives. <\/p>\n\n\n\n
When the ROI goes out of control, it should set alarm bells ringing. <\/p>\n\n\n\n
Therefore, coming back to our story, I would be losing sleep over the Fire Phone if I were Jeff Bezos. Why wasn\u2019t he? The answer, and I quote him, came in a 2018 letter to Amazon\u2019s shareholders:<\/p>\n\n\n\n
\u201cAs a company grows, everything needs to scale, including the size of your failed experiments. If the size of your failures isn\u2019t growing, you\u2019re not going to be inventing at a size that can actually move the needle.\u201d<\/em><\/p>\n\n\n\n Great story, right?<\/p>\n\n\n\n Unfortunately though, if your manager isn\u2019t Bezos, and your company doesn\u2019t have deep pockets like Amazon, you\u2019re most likely in trouble. <\/p>\n\n\n\n Your manager might ask you questions like \u201cDid the variation win?\u201d or \u201cDid this experiment move the needle enough to put it in my town hall presentation?\u201d. If the answer to these questions is NO, the experimentation exercise<\/a> is termed futile or having no ROI. <\/p>\n\n\n\n Why so, you ask? This is because there is so much literature around us which declares uplift (in conversion rates or revenues) as the only metric to measure the success of experimentation. Everyone expects an uplift from experiment, and then terms the uplift as either \u2018present\u2019 or \u2018absent\u2019 – there is no in-between. <\/p>\n\n\n\n This approach reduces experimentation to a point in a checklist<\/a> which companies want to tick off. This is because they see experimentation as the goose, which will suddenly lay golden eggs (read: uplift). <\/p>\n\n\n\n The truth is that though the industry has conditioned us to think this way, this should not be the reasoning behind any experiment.<\/p>\n\n\n Essentially, experiments help in achieving the following objectives: <\/p>\n\n\n Read that again. It\u2019s not \u2018correct\u2019 or \u2018right\u2019; it\u2019s making \u2018better\u2019 decisions. Experimentation aids data-driven decision making<\/a>, which is better than based on gut or instinct. Without an experiment, there is no way to objectively denounce an idea.<\/p>\n\n\n It\u2019s always better to test your hypotheses<\/a> than doing a full rollout of your new website or product features. This, in turn, minimizes the risk of impacting all your current business metrics – whether it\u2019s the conversion rate on your website or feature adoption rates.<\/p>\n\n\n Building an experimentation roadmap<\/a> helps you prioritize and differentiate between the must-haves and good-to-haves. Some experiments may not give you an uplift but will provide you tonnes of learnings for your next one.<\/p>\n\n\n The problem arises when \u2018HiPPOs\u2019 decide which experiment to run first, and their primary expectation is a revenue uplift. The fundamental purpose of experimentation to make better decisions goes for a toss. <\/p>\n\n\n Imagine a situation where you, as a CRO practitioner, have been running an experimentation program for a quarter or 6 months with multiple experiments across different pages of your website. Most of these tests have yielded statistically significant winners, and you are elated. <\/p>\n\n\n\n However, when you look at the overall conversion rate of your website, it is almost the same as a quarter before. What do you tell your manager? How do you justify the investments you made in the CRO product which you championed to buy? Does it mean your optimization initiatives are worthless? (Story of your life? Shoot me an email<\/a> or tag me on Twitter<\/a> if I nailed it.)<\/p>\n\n\n\n Imagine the same situation–with a quarter spent in testing and continuous optimization–but this time your overall revenue has shot up by 3%. You are elated as this is your moment. But as soon as you enter your CEO\u2019s cabin\/zoom room, you see the head of SEO, head of performance marketing, and others already showing how they are responsible for this uplift. <\/p>\n\n\n\n It makes you wonder and doubt how you can isolate the impact of the CRO experiments<\/a> from other variables and proclaim this as a victory for you and your team.<\/p>\n\n\n\n (Sidenote: we organized a webinar to try and unknot the mystery behind attribution challenges with experimentation<\/a>.)<\/p>\n\n\n\n It is hard to accurately calculate the ROI of your experimentation program because of issues in forecasting, dealing with multiple variables, and running multiple experiments. CRO practitioners deal in averages, and what might seem an impact \u2018wave\u2019 for a specific segment of users might just be a ripple in the ocean when you look at global goals.<\/p>\n\n\n\n And you\u2019re not alone. Almost 53% of CRO professionals cannot calculate ROI of experimentation<\/a>.<\/p>\n\n\n\n Instead of putting resources behind chasing that one ROI number, I would advise using those resources to increase the testing velocity. This is because the ROI on experimentation is amplified by the customer insights you gain with each experiment<\/a> you perform. That\u2019s what Bezos did with Ian Freed. <\/p>\n\n\n\n The voice recognition software of the Fire Phone could follow commands and fetch information from the cloud. This feature was talked about and loved by the customers. It made Bezos curious, and he put Freed on a project to build a team and technology to respond to voice commands. Four months later, Echo was born. Jeff Bezos looked back at the incident and remarked: <\/p>\n\n\n\n \u201cWhile the Fire Phone was a failure, we were able to take our learnings (as well as the developers) and accelerate our efforts building Echo and Alexa.\u201d<\/em><\/p>\n\n\n\n Think about the players who have disrupted industries – Netflix, Amazon, Booking, etc. The things that they have in common are a company-wide culture of experimentation and unmatched testing velocity that allows them to fail fast and move on to the next innovation. <\/p>\n\n\n\n \u201cIf you have to kiss a lot of frogs to find a prince, find more frogs and kiss them faster and faster.\u201d ~Mike Moran<\/em><\/p>\n\n\n
Image source: Geekwire<\/a><\/figcaption><\/figure>\n\n\n\nWhy do we experiment?<\/h2>\n\n\n
1. Making better decisions<\/strong><\/h3>\n\n\n
2. Reducing ambiguities and risk of losing business<\/strong><\/h3>\n\n\n
3. Prioritizing and learning what works and what does not<\/strong><\/h3>\n\n\n
Image source: Medium<\/a>
<\/figcaption><\/figure>\n<\/div>\n\n\n3. Prioritizing and learning what works and what does not<\/strong><\/h3>\n\n\n
Image source: Marketingsherpa<\/a><\/figcaption><\/figure>\n\n\n\nDownload Free: A\/B Testing Guide<\/span><\/a><\/em><\/h2>\n\n\n
So what is the impact of experimentation?<\/h2>\n\n\n
<\/figure>\n\n\n1. Innovation\/Breakthrough<\/strong><\/h3>\n\n\n
2. Reducing the risk of new launches<\/strong><\/h3>\n\n\n