{"id":78698,"date":"2023-04-11T11:15:23","date_gmt":"2023-04-11T05:45:23","guid":{"rendered":"https:\/\/vwo.com\/blog\/?p=78698"},"modified":"2025-05-05T14:44:15","modified_gmt":"2025-05-05T09:14:15","slug":"requirements-for-running-experimentation-programs","status":"publish","type":"post","link":"https:\/\/vwo.com\/blog\/requirements-for-running-experimentation-programs\/","title":{"rendered":"Requirements for Running Serious Experimentation Programs"},"content":{"rendered":"\n

Haley is the founder of Chirpy<\/a> and specializes in user research-based experimentation<\/em>. <\/p>\n\n\n\n

Running experimentation programs<\/a> is an art and a science. I say it all the time. Programs should have some level of rigor \u2013 meaning systems, processes, and procedures. It\u2019s not something to take lightly. Believing that anyone can start a program tomorrow with minimal preparation and planning is a mistake. Unfortunately, though, that happens all the time. It leads to lots of money, time, and effort wasted \u2013 unsurprisingly. This leads me to the topic of preparation.<\/em>\u00a0<\/p>\n\n\n\n

If you want to be serious about experimentation and level up how competitive you are in the marketplace, you better be doing it well. You should assume your competitors are doing it well. So if this resonates with you, continue reading and I guarantee you\u2019ll pick up a golden nugget or two to use immediately.<\/p>\n\n\n\n

\n
\"Feature<\/figure>\n<\/div><\/figure>\n\n\n\n

The unavoidable precursor to building an experimentation program that will make or break you: Pre-test calculations<\/h2>\n\n\n\n

Pre-test calculations. Ever heard of them? Have you done them? Does MDE or minimum detectable effect sound familiar? How about duration estimates<\/a> or sample sizes? I hope you know what I\u2019m talking about although I\u2019d bet money that a large majority of you don\u2019t \u2013 simply because of my own personal experience with clients.<\/p>\n\n\n\n

Before you do anything related to experimentation, please see if you have enough data volume to do it. See if you\u2019re able to test at all through pre-test calculations. By data volume, I mean visitors and conversions. Visitors can be whatever you typically use (e.g., sessions, users, MAUs, etc.). Conversions are from the primary metric you\u2019re going to use in your tests. Know this:<\/p>\n\n\n\n

    \n
  1. Not every business has enough data volume to do experimentation at any capacity.<\/li>\n\n\n\n
  2. If you can do it, know that you don’t just pick your desired velocity out of thin air. It’s based on calculations.<\/li>\n<\/ol>\n\n\n\n

    The #1 culprit for ignoring one or both of these points: sales people. If you\u2019re looking at buying any kind of tool, make sure this is part of the conversation. The minimum barrier to entry to have an experimentation program: enough data volume to run one test within eight or fewer weeks in one swimlane.<\/p>\n\n\n\n

    I covered this topic in detail a few months ago for Experiment Nation<\/a>. Know that if you don’t understand this topic and do it from day one, it will haunt you and definitely cause undesired outcomes of some kind eventually. One other very important note: know if your testing tool (or the one you plan to use) is built based on fixed-horizon testing or sequential testing. This affects the calculations and how you run your program.<\/p>\n\n\n\n

    Step 1 (Post-precursor): Measurement & data quality<\/h2>\n\n\n\n

    If you\u2019ve cleared the pre-test calculations hurdle and you\u2019ve confirmed that you do have enough data volume to test, the next hurdle to moving forward is measurement and data quality. You must<\/em> know what you\u2019re aiming at in this work; otherwise, you\u2019ll flounder like a fish on a riverbank. Too many teams don\u2019t know what they\u2019re working toward \u2013 like form submissions, transactions, revenue, LTV, etc. <\/p>\n\n\n\n

    Understand what your primary, secondary, and tertiary metrics are for experimentation and the business as a whole. Understand it with complete clarity. Don\u2019t allow for lingering confusion or uncertainty. Make sure everyone<\/em> is on the same page.<\/p>\n\n\n\n

    Then, once you have that much, make sure you\u2019re collecting that data in the right places and<\/em> that you can trust it<\/a>. <\/p>\n\n\n\n

    If measurement and\/or data quality are disasters, just stop. Stop everything and devote all of your efforts toward getting it right. Think of experimentation as a pyramid. These two things are the foundational layers of the pyramid. If it cracks at any point in time, everything else will crumble on top of it. I promise.<\/p>\n\n\n\n

    I\u2019ll say that I know these can be hard. Getting them right can take extra time. Maybe even more than a month or two. Getting them right is worth it though. I\u2019ve seen issues come up six months or more after launching a program \u2013 only for everything to eventually come to a screeching halt. No one is happy at that point.<\/p>\n\n\n\n

    A note on what a primary metric should be\u2026<\/p>\n\n\n\n

    This is a divisive topic among practitioners sometimes. I have a very firm stance on the matter, specifically when it comes to marketing teams and websites (not necessarily product teams and products).<\/p>\n\n\n\n

    Primary metrics should always be down-funnel metrics. Orders. Form submissions. MQLs. Revenue. LTV. SQLs. You get the idea. Some people say they should always be the action closest to the change you\u2019re making or engagement metrics. Wrong. No. Nope. Incorrect. BS. Whoever tells you this should be the one to have to justify the program in six months or a year to the CMO or CEO of the company. They\u2019ll be in the hot seat. DO NOT have a program full of tests focused on button clicks, click-throughs, pageviews, avg. session duration, exit rate, bounce rate, video views, and so on and so forth. That\u2019s not going to justify the thousands or hundreds of thousands of dollars spent to do this work. Everyone wants to know their ROI and how the work impacted the bottom line. Button clicks aren\u2019t going to do that. <\/p>\n\n\n\n

    I\u2019m not saying don\u2019t measure engagement metrics or higher-funnel metrics, but they should be secondary or tertiary metrics. Not primary ones. They add context to the story of a test. They\u2019re not what tests are hinged on when the time comes to make a decision. Note, I\u2019m also not saying there are never exceptions. Still evaluate tests on a case-by-case basis.<\/p>\n\n\n\n

    A word of advice: To those debating this topic amongst yourselves, I always tell teams to discuss the options and decide for themselves. Just make sure you come to a collective conclusion that everyone abides by moving forward.<\/p>\n\n\n\n

    Step 2: User research & ideation<\/h2>\n\n\n\n

    At this point, you should (1) know you have enough data volume to test and (2) know what you\u2019re measuring and that you\u2019re collecting proper data that you can trust. So what\u2019s next? It\u2019s coming up with what to test. What are your test ideas? How are you going to generate them?<\/p>\n\n\n\n

    Guess what most teams do? They go off of gut feelings and a lot of \u201cwe think,\u201d \u201cwe feel,\u201d and \u201cwe believe.\u201d That is far too subjective, and it\u2019s a terrible way to run a program. That approach is not data-backed at all. It\u2019s what practitioners call \u201cspaghetti testing\u201d AKA flinging stuff at the wall and hoping it sticks. Data-based conversations don\u2019t involve much of that sort of language, and the data that\u2019s needed comes from user research. I get asked what \u201cresearch\u201d means all the time.<\/p>\n\n\n\n

    Well, there are several methodologies that collect data including but not limited to analytics, polls, surveys<\/a>, user testing, message testing, heatmaps<\/a>, session recordings<\/a>, card sorting, tree testing, customer journey mapping<\/a>, personas, and many more. There are also several tools to help us complete each of these. I always say to start with one or two and work your way to other ones from there. That\u2019s certainly better than nothing. Technically, I don\u2019t really count analytics anymore because every company has analytics data these days. If you don\u2019t have that, you likely have bigger fish to fry. If you do have it though, strive for one or two beyond that even (and don\u2019t say \u201coh we\u2019re good then\u201d).<\/p>\n\n\n\n

    There is a methodology called heuristic evaluation. That\u2019s when someone visually assesses an experience and develops insights based on their experience and expertise. There\u2019s a time and a place for it but it\u2019s not backed by \u201chard data\u201d most of the time. It\u2019s quite subjective and will be different to some extent depending on who completes it. Know your program shouldn\u2019t be based on these types of insights.<\/p>\n\n\n\n

    I’m not going to cover how to do research in great detail here, but you can check out one of my VWO Webinars here where I talk more about CXL’s ResearchXL model.<\/p>\n\n\n\n

    \n