A/B Testing Guide
Everything you need to know about A/B Testing - from why to how, challenges to examples
What is A/B Testing?
A/B testing (also known as split testing) is a process of showing two variants of the same web page to different segments of website visitors at the same time and comparing which variant drives more conversions.
In the online world, the number of visitors on your website equals the number of opportunities you have to expand your business by acquiring new customers and build relationships by catering to existing ones. It is your conversion funnel that decides whether your website gets good traffic and converts more visitors.
Businesses want visitors to take action (also called a conversion) on their website, and the rate at which a site can drive this is called its “conversion rate.” The more optimized your funnel, the higher is the conversion rate. One of the most important ways to optimize your website’s funnel in digital marketing is A/B testing.
Typically in A/B testing, the variant that gives higher conversions is the winning one, and that variant can help you optimize your site for better results.
The metrics for conversion are unique to each website. For eCommerce, it may be the sale of the products, while for B2B, it may be the generation of qualified leads. A/B testing is one of the components of the overarching process of Conversion Rate Optimization (CRO). You can gather qualitative and quantitative user insights and use them to understand your potential customers and optimize your conversion funnel based on that data.
Why Should You A/B Test?
If B2B businesses today are unhappy with all the unqualified leads they get per month, eCommerce stores, on the other hand, are struggling with a high cart abandonment rate. Meanwhile, media and publishing houses are also dealing with low viewer engagement. These core conversion metrics are affected by some common problems like leaks in the conversion funnel, drop-offs on the payment page, etc.
Let’s see why you should do A/B testing to deal with all these problems:
Solve Visitor Pain Points
Visitors on your website come to achieve a specific goal that they have in mind. It may be to understand more about your product or service, buy a product, read/learn more about a particular topic, or simply browse. Whatever the user’s goal, they may face some common pain points while achieving their goal: it can be a confusing copy or hard to find the CTA button like buy now, request a demo, etc.
Not being able to achieve their goals leads to a bad user experience. This increases friction and eventually impacts your conversion rates. Use data gathered through visitor behavior analysis tools such as heatmaps, Google Analytics, and website surveys to solve your visitors’ pain points. This stands true for all businesses, be it eCommerce, travel, SaaS, education, or media and publishing.
Get Better ROI from Existing Traffic
As most marketers have come to realize, the cost of acquiring any quality traffic can be huge. A/B testing lets you make the most out of your existing traffic and helps you increase conversion without having to spend on acquiring new traffic. A/B testing can give you high ROI as sometimes, even the most minor changes can result in a significant increase in conversions.
Reduce Bounce Rate
One of the most important metrics to be tracked to judge your website’s performance is its bounce rate. There may be many reasons for your website’s high bounce rate, such as too many options, expectations mismatch, and so on. As different websites serve different goals and cater to different audiences, there is no sure-shot way of fixing the bounce rate. One way to do it is through A/B testing. With A/B testing, you can test multiple variations of your website’s element till you find the best possible version. This improves your user experience, making visitors spend more time on your site and reduce bounce rates.
Make Low-risk Modifications
Make minor, incremental changes to your web page with A/B testing instead of getting the entire page redesigned. This can reduce the risk of jeopardizing your current conversion rate. A/B testing lets you target your resources for maximum output with minimal modifications, resulting in increased ROI. An example of that could be product description changes. You can perform an A/B test when you plan to remove or update your product descriptions. You do not know how your visitors will react to the change.
A/B testing is one way to ascertain which side the weighing scale will tilt. Another example of low-risk modification can be the introduction of a new feature change. Before introducing a new feature, launching that new feature as an A/B test in the web page’s copy can make the outcome much more predictable. It can be very useful beneficial if the changes affect customer data or purchase funnel. Changes without testing may or may not pay off. Testing and then making changes can make the outcome certain.
Achieve Statistically Significant Improvements
A/B testing is completely data-driven with no room for guesswork, gut feelings, or instincts. You can easily determine a “winner” and a “loser” based on statistically significant improvements on metrics like time spent on the page, number of demo requests, cart abandonment rate, click-through rate, and so on.
Profitably Redesign your Website
Redesigning can range from a minor text or color tweak in a CTA button to completely revamping a webpage. The decision to implement one version or the other should always be based on data-driven A/B testing. Do not quit testing with the design being finalized. As the new version goes live, test other web page elements to ensure that the most engaging version is visible to the visitors.
How do you Perform an A/B Test?
A/B testing offers a very systematic way of finding out what works and what doesn’t work in any given marketing campaign. Most marketing efforts are geared toward driving more traffic. As traffic acquisition becomes more difficult and expensive, it becomes paramount to offer your users the best experience who comes to your website. This will help them achieve their goals and convert in the fastest and most efficient manner possible. A/B testing in marketing allows you to make the most out of your existing traffic.
A structured A/B testing program can make marketing efforts more profitable by pinpointing the most crucial problem areas that need optimization. A/B testing is now moving away from being a standalone activity conducted once in a blue moon to a more structured and continuous activity, which should always be done through a well-defined CRO process. Broadly, it includes the following steps:
Step 1: Research
Before building an A/B testing plan, one needs to conduct thorough research on how the website is currently performing. You will have to collect data on everything related to how many users are coming onto the site, which pages drive the most traffic, what are the various conversion goals of different pages etc. The A/B testing tools used here can include quantitative website analytics tools such as Google Analytics, Omniture, Mixpanel, etc. These help figure out your most visited pages, pages with the most time spent, or pages with the highest bounce rate. For example, you may want to start by shortlisting pages with the highest revenue potential or the highest daily traffic. Following this, you may want to dive deeper into the qualitative aspects of this traffic.
Heatmap tools like VWO’s AI-powered free heatmap generator are the leading technology used to determine where users are spending the most time on, their scrolling behavior, etc. This can help you identify problem areas on your website. Another popular tool used to do more insightful research is website user surveys. Surveys can act as a direct conduit between your website team and the end-user and often highlight issues that may get missed in aggregate data. Further, you can derive qualitative insights from session recording tools that collect visitor behavior data to identify gaps in the user journey. In fact, session recording tools combined with form analysis surveys can uncover insights on why users may not fill your form. It may be due to some fields that ask for personal information or users, maybe abandoning your forms for too long.
As we can see, both quantitative and qualitative research can help us prepare for the next step in the process, making actionable observations for the next steps.
Step 2: Observe and Formulate Hypothesis
Get closer to your business goals by logging research observations and creating data-backed hypotheses aimed at increasing conversions. Without these, your test campaign is like a directionless compass. The qualitative and quantitative research tools can only help you with gathering visitor behavior data. It is now your responsibility to analyze and make sense of that data. The best way to utilize every bit of data collated is to analyze it, make keen observations on them, and then draw website and user insights to formulate data-backed hypotheses. Once you have a hypothesis ready, test it against various parameters like how much confidence you have of it winning, its impact on macro goals, how easy it is to set up, etc.
Step 3: Create Variations
The next step in your testing program should be to create a variation based on your hypothesis and A/B test it against the existing version (control). A variation is another version of your current version with changes that you want to test. You can test multiple variations against the control to see which one works best. Create a variation based on your hypothesis of what might work from a UX perspective. For example, enough people not filling forms? Does your form have too many fields? Does it ask for personal information? Maybe you can try a variation with a shorter form or another variation by omitting fields that ask for personal information.
Step 4: Run Test
Before we get to this step, let’s first explore how many kinds of testing methods are there and when to use which method. A/B Testing, Multivariate Testing, Split URL Testing, and Multipage Testing are four different types of testing. We have already discussed the first kind, namely A/B testing. Let us now move on to the other types of testing.
Split URL Testing
Many people confuse Split URL Testing with A/B testing, but the two are fundamentally different. Split URL Testing is testing multiple versions of your webpage hosted on different URLs. Your website traffic is split between the control and variations, and respective conversion rates are measured to decide the winning version. The main difference between a Split URL test and an A/B test is that the variations are hosted on different URLs in case of a Split URL test.
A/B testing is preferred when only front-end changes are required, but split URL testing is preferred when significant design changes are necessary and don’t want to touch the existing website design.
Creating a Split URL test broadly consists of the following steps:
- Setting up pages for the Split URL test
- Adding conversion goals and estimating test duration
- Finalizing the test
- Previewing and starting the test
Multivariate Testing (MVT)
Multivariate testing is a method in which changes are made to multiple sections of a webpage, and variations are created for all the possible combinations. In this testing method, you can test all the combinations within a single test. The multivariate test helps you figure out which element on a web page makes the most impact on its conversion rate. It is more complicated than A/B testing and is best suited for advanced marketing professionals.
For instance, you decide to test two versions of each hero image and the CTA button color on a webpage. By using MVT, you can create one variation for the hero image and one for the CTA button color. To test all the versions, create combinations of all the variations, as shown in the image below.
After running a multivariate test on all combinations, you can use the data procured to determine which combination affects your page’s conversion rate most and deploy that combination.
Multipage testing is a form of experimentation where you can test changes to particular elements across multiple pages.
There are two ways of doing multipage testing. One, you can either take all the pages of your sales funnel and create new versions of each, which makes your challenger sales funnel, and you then test it against the control. This is called Funnel Multipage testing. And two, you can test how the addition or removal of the recurring element(s), like security badges, testimonials, and so on, can impact conversions across an entire funnel. This is called Classic or Conventional Multipage testing.
In addition to the four types just discussed, there are also two different statistical approaches to testing: Frequentist and Bayesian.
The frequentist approach of probability defines the probability of an event with relation to how frequently (hence the name) a particular event occurs in a large number of trials/data points. When applied to the world of A/B testing, one can see that anyone going with the frequentist approach would need more data (a function of more number of visitors tested and over longer durations) to come to the right conclusions. This is something that limits you in scaling up any A/B testing effort.
According to the Frequentist approach, it is essential to define your A/B test’s duration based on sample size to reach the right test conclusions. The tests are based on the fact that every experiment can be repeated infinite times.
Following this approach calls for a lot of attention to detail for every test that you run because for the same set of visitors, you’ll be forced to run longer duration tests than the Bayesian approach. Hence, each test needs to be treated with extreme care because there are only a few tests that you can run in a given timeframe. Unlike Bayesian statistics, it is less intuitive and often proves difficult to understand.
On the other hand, Bayesian statistics is a theory based on the Bayesian interpretation of probability, where probability is expressed as a degree of belief in an event. In simple words, the more you know about an event, the better and faster you can predict the end outcomes. Rather than being a fixed value, probability under Bayesian statistics can change as new information is gathered. This belief may be based on past information such as the results of previous tests or other information about the event.
Unlike the frequentist approach, the Bayesian approach provides actionable results almost 50% faster than the older Frequentist method while focusing on statistical significance. At any given point, provided you have enough data at hand, the Bayesian approach tells you the probability of variation A having a lower conversion rate than variation B or the control. It does not have a defined time limit attached to it, nor does it require you to have an in-depth knowledge of statistics.
In the simplest of terms, the Bayesian approach is akin to how we approach things in everyday life. For instance, you misplaced your mobile phone in your house. As a frequentist, you would only use a GPS tracker to track it and only check the area the tracker is pointing to. While as a Bayesian, you will use a GPS tracker and check all the places in the house you earlier found your misplaced phone. In the former, the event is considered a fixed value, while in the latter, all past and future knowledge is utilized to locate the phone. To get a clearer understanding of the two, take a look at the comparison below:
|Frequentist Approach||Bayesian Approach|
|Frequentist Statistics follow the ‘Probability as Long-Term Frequency’ definition of probability.||Bayesian Statistics follow the notions of ‘Probability as Degree of Belief’ and ‘Logical Probability’|
|In this approach, you only use data from your current experiment. The frequentist solution is to conduct tests and draw conclusions.||In this approach, you use your prior knowledge gathered from the previous experiments and try to incorporate that information into your current data. The Bayesian solution is to use existing data to draw conclusions.|
|Give an estimated mean (and standard deviation) of samples where A beats B but completely ignores the cases when B beats A.||Takes into account the possibility of A beating B and also calculates the range of the improvement you can expect.|
|Requires the test to run for a set period of time to get correct data from it but can’t figure out how close or far A and B actually are. It fails to tell you the probability of A beating B.||Gives you more control over testing. You can now plan better, have a more accurate reason to end tests, and get into the nitty-gritty of how close or far apart A and B are.|
Once you have locked down on either one of these types and approaches based on your website’s needs and business goals, kick off the test and wait for the stipulated time for achieving statistically significant results. Keep one thing in mind – no matter which method you choose, your testing method and statistical accuracy will determine the end results. For example, one such condition is the timing of the test campaign. The timing and duration of the test have to be on point. Calculate the test duration keeping in mind your average daily and monthly visitors, estimated existing conversion rate, minimum improvement in conversion rate you expect, number of variations (including control), percentage of visitors included in the test, and so on.
Use our Bayesian Calculator to calculate the duration you should run your A/B tests to achieve statistically significant results.
Step 5: Result Analysis and Deployment
Even though this is the last step in finding your campaign winner, analysis of the results is extremely important. Because A/B testing calls for continuous data gathering and analysis, it is in this step that your entire journey unravels. Once your test concludes, analyze the test results by considering metrics like percentage increase, confidence level, direct and indirect impact on other metrics, etc. After you have considered these numbers, if the test succeeds, deploy the winning variation. If the test remains inconclusive, draw insights from it, and implement these in your subsequent tests.
A/B testing lets you systematically work through each part of your website to improve conversions. Use VWO’s holistic experimentation suite and convert every test into successful learning.
What Can you A/B Test?
Your website’s conversion funnel determines the fate of your business. Therefore every piece of content that reaches your users via your website must be optimized to its maximum potential. This is especially true for elements that can influence visitor behavior and conversion rate. When undertaking an optimization program, the following key elements should be A/B tested (the list, however, is not exhaustive):
1. Headlines and Sub-headlines
Your headline is the first thing that visitors see on your page. The headline is what defines your first impression in a visitor’s eyes. And the first impression is what determines if the visitor will go further down your conversion funnel. This is why it becomes imperative to be extra cautious about your website’s copy, writing style, and formatting. Make sure your headline catches the visitors’ attention as soon as they land on your website. Keep it short and to the point, ensuring it talks clearly about what your product or service is and its benefits. Try A/B testing various fonts, sizes, copy, and messaging.
The body of your website should clearly state what the visitor is getting – what’s in store for them. It should also resonate with your page’s headline. A well-written headline and body can increase the chances of turning your website into a conversion magnet. While writing content for your page’s body, keep in mind these two parameters:
- Writing style: Use the right tonality based on the target audience. Your copy should directly address the end-user and answer all their questions. It should consist of key phrases that improve usability and stylistic elements that highlight important points.
- Formatting: Use relevant headlines and subheadlines, break the copy into small and easy paragraphs, and format it for skimmers using bullet points or lists.
Design and Layout
Because everything seems so essential, businesses sometimes struggle with finding only the most essential elements to keep on their website. With A/B testing, this problem can be solved once and for all. For example, as an eCommerce store, your product page is extremely important from a conversion perspective. One thing for sure is that with technological progress in its current stage, customers like to see everything in high definition before buying it. Therefore, your product page must be in its most optimized form in terms of design and layout. Along with the copy, the design and layout of a page include images (product images, offer images, etc.) and videos (product videos, demo videos, advertisements, etc.). Your product page should answer all of your users’ questions without confusing them and without getting cluttered:
- Provide clear information: Based on the products you sell, find creative ways to provide all necessary context and accurate product descriptions so that prospective buyers do not get overwhelmed with an unorganized copy while looking for answers to their queries. Write clear copies and provide easily noticeable size charts, color options, etc.
- Highlight customer reviews: Add both good and bad reviews for your products. Negative reviews add credibility to your store.
- Write simple content: Avoid confusing potential buyers with complicated language in the quest to decorate your content. Keep it simple and fun to read.
- Create a sense of urgency: Add tags like ‘only 2 left in stock’, countdowns like ‘offer ends in 2hours 15 minutes’, highlight exclusive discounts and festive offers, etc., to nudge the prospective buyer to purchase immediately.
Other important pages whose design needs to be on point are pages like the home page and landing page. Use A/B testing to discover the most optimized version of these critical pages. Test many ideas like adding plenty of white space and high definition images, featuring product videos instead of images, or testing out different layouts. Declutter your pages using insights from click maps to analyze dead clicks and identify distractions. The less cluttered your home page and landing pages, the more likely visitors can easily and quickly find what they are looking for.
Another element of your website that you can optimize by A/B testing is your website’s navigation. It is the most crucial element when it comes to delivering an excellent user experience. Ensure you have a clear plan for your website’s structure and how different pages will be linked to each other and react within that structure.
Your website’s navigation starts on the home page. The home page is the parent page from which all other pages emerge and links all the pages. Ensure your structure is such that visitors quickly find what they are looking for and do not get lost because of broken navigation. Each click should direct visitors to the desired page.
Here are some ideas that can help you up to your navigation game:
- Match visitor expectations by placing your navigation bar in standard places like horizontal navigation on the top and vertical down the left to make your website easier to use.
- Make your website’s navigation predictable by keeping similarly themed content in the same bucket or in related buckets to reduce your visitor’s cognitive load. For example, as an eCommerce store, you may be selling a variety of earphones and headphones. Some of them may be wired, while others may be wireless or ear-pods. Bucket these in such a way that when a visitor looks for earphones or headphones, they find all these varieties in one place rather than having to search for each kind separately
- Creating a fluid, easy-to-navigate website by keeping the structure simple, predictable, and matching your visitors’ expectations can increase the chances of uplifting your conversion rate and create a delightful customer experience forcing visitors to come back to your website.
Forms are mediums through which prospective customers get in touch with you. They become even more important if they are part of your purchase funnel. Just as no two websites are the same, no two forms addressing the different audience are the same. While for some businesses, a small comprehensive form may work, for many others, long forms might do wonders for their lead quality. You can figure out which style works for your audience the best by using research tools/methods like form analysis to determine the problem area in your form and work towards optimizing it.
CTA (Call To Action)
The CTA is where all the real action takes place – whether or not visitors finish their purchases and convert if they fill out the sign-up form or not, and more such actions that have a direct bearing on your conversion rate. With A/B testing, you can A/B test different copies, placement, colors & sizes, etc., for your CTA till you find the winning variation – and then test the winning version further to optimize it even more.
Social proof may take the form of recommendations and reviews from experts of the particular fields, from celebrities and customers themselves, or can come as testimonials, media mentions, awards and badges, certificates, and so on. The presence of these proofs validates the claims made by your website. A/B testing can help you determine if adding social proof is a good idea, what kinds of social proof you should use, and how many you should add. You can test different uses of social proof, layout, and placement.
What are the Mistakes to Avoid While A/B Testing?
Mistake #1: Not Planning your Optimization Roadmap
- Invalid hypothesis: In A/B testing, a hypothesis is formulated before conducting a test. All the next steps depend on it: what should be changed, why should it be changed, what the expected outcome is, and so on. If you start with the wrong hypothesis, the probability of the test succeeding decreases.
- Taking others’ word for it: Sure, someone else changed their sign-up flow and saw a 30% uplift in conversions. But it is their test result, based on their traffic, their hypothesis, and their goals. Here’s why you should not implement someone else’s test results as is onto your website: no two websites are the same – what worked for them might not work for you. Their traffic will be different; their target audience might be diverse; their optimization method may have been different than yours, and so on.
Mistake #2: Testing too Many Elements Together
Industry experts caution against running too many tests at the same time. Testing too many elements of a website together makes it difficult to pinpoint which element influenced the test’s success or failure most. Apart from this, the more the elements tested, the more needs to be traffic on that page to justify statistically significant testing. Thus, prioritization of tests is indispensable for successful A/B testing.
Mistake #3: Ignoring Statistical Significance
If gut feelings or personal opinions find a way into hypothesis formulation or while you are setting the A/B test goals, it is most likely to fail. Irrespective of everything, whether the test succeeds or fails, you must let it run through its entire course so that it reaches its statistical significance.
For a reason, that test results, no matter good or bad, will give you valuable insights and help you plan your upcoming test in a better manner.
You can get more information about the different types of errors while dealing with the maths of A/B testing.
Mistake #4: Using Unbalanced Traffic
Businesses often end up testing unbalanced traffic. A/B testing should be done with the appropriate traffic to get significant results. Using lower or higher traffic than required for testing increases the chances of your campaign to fail or generate inconclusive results.
Mistake #5: Testing for Incorrect Duration
Based on your traffic and goals, run A/B tests for a certain period until it achieves statistical significance. Running a test for too long or too short a period can result in the test failing or producing insignificant results. Because one version of your website appears to be winning within the first few days of starting the test does not mean that you should call it off before time and declare a winner. Letting a campaign run for too long is also a common blunder that businesses commit. The duration for which you need to run your test depends on various factors like existing traffic, existing conversion rate, expected improvement, etc.
Learn how long you should run your test.
Mistake #6: Failing to Follow an Iterative Process
A/B testing is an iterative process, with each test building upon the results of the previous tests. Businesses give up on A/B testing after their first test fails. But to improve the chances of your next test succeeding, you should draw insights from your last tests while planning and deploying your next test. This improves the probability of your test succeeding with statistically significant results.
Additionally, do not stop testing after a successful one. Test each element repetitively to produce the most optimized version of it, even if they are a product of a successful campaign.
Mistake #7: Failing to consider external factors
Tests should be run in comparable periods to produce meaningful results. It is wrong to compare website traffic on the days when it gets the highest traffic to the days when it witnesses the lowest traffic because of external factors such as sales, holidays, and so on. Because the comparison here is not made between likes, the chances of reaching an insignificant conclusion increase. Use VWO’s A/B Test Significance Calculator to know if the results your test achieved were significant or not.
Mistake #8: Using the Wrong Tools
With A/B testing gaining popularity, multiple low-cost tools have also come up. Not all of these tools are equally good. Some tools drastically slow down your site, while others are not closely integrated with necessary qualitative tools (heatmaps, session recordings, and so on), leading to data deterioration. A/B testing with such faulty tools can risk your test’s success from the start.
Mistake #9: Sticking to Plain Vanilla A/B Testing Method
While most marketers recommend that you start your experimentation journey by running small A/B tests on your website to get the hang of the entire process but, in the long run, sticking to the plain vanilla A/B testing method won’t work wonders for your organization. For instance, if you plan to revamp one of your website’s pages entirely, you ought to make use of split testing. Meanwhile, if you wish to test a series of permutations of CTA buttons, their color, the text, and the image of your page’s banner, you must use multivariate testing.
What are the Challenges of A/B Testing?
The ROI from A/B testing can be huge and positive. It helps you direct your marketing efforts to the most valuable elements by pinpointing exact problem areas. But every once in a while, as a marketer, you may face some challenges when deciding to undertake A/B testing. The 6 primary challenges are as follows:
Challenge #1: Deciding What to Test
You can’t just wake up one day and decide to test certain elements of your choice. A bitter reality that marketers are now realizing is that not all small changes that are easy to implement are always the best when considering your business goals and often fail to prove significant. The same goes for complex tests. This is where website data and visitor analysis data come into play. These data points help you overcome the challenge of ‘not knowing what to test’ out of your unending backlog by generally pointing to the elements which may have the most impact on your conversion rates or by directing you to pages with the highest traffic.
Challenge #2: Formulating Hypotheses
In great resonance with the first challenge is the second challenge: formulating a hypothesis. This is where the importance of having scientific data at your disposal comes in handy. If you are testing without proper data, you might as well be gambling away your business. With the help of data gathered in the first step (i.e., research) of A/B testing, you need to discover where the problems lie with your site and come up with a hypothesis. This will not be possible unless you follow a well-structured and planned A/B testing program.
Challenge #3: Locking in on Sample Size
Not many marketers are statisticians. We often make the mistake of calling conclusive results too quickly because we are more often than not after quick results. As marketers, we need to learn about sample sizes, particularly how large our testing sample size should be based on our web page’s traffic.
Challenge #4: Analyzing Test Results
With A/B testing, you will witness success and failure at each step. This challenge, however, is pertinent to both successful and failed tests:
- Successful campaigns: Great that you ran two tests, and both of them were successful in producing statistically significant results. What next? Yes, deploying the winner, but what after that? What marketers often fail to do or find difficult is interpreting test results. Interpreting test results after they conclude is extremely important to understand why the test succeeded. A fundamental question to be asked is – why? Why did customers behave the way they did? Why did they react a certain way with one version and not with the other versions? What visitor insights did you gather, and how can you use them? Many marketers often struggle or fail to answer these questions, which not only help you make sense of the current test but also provide inputs for future tests.
- Failed campaigns: Sometimes, marketers don’t even look back at failed tests. They either have a hard time dealing with them, such as telling the team about the failed tests or having no clue what to do with them. No failed test is unsuccessful unless you fail to draw learnings from them. Failed campaigns should be treated like pillars that would ultimately lead you to success. The data gathered during the entire A/B testing process, even if in the end, the test failed, is like an unopened pandora box. It contains a plethora of valuable data and insights that can give you a head-start for your next test.
Additionally, with the lack of proper knowledge on how to analyze the gathered data, the chances of data corruption increase manifold. For example, there’s no end to scrolling through heatmaps data or sessions recording data if you do not have a process in place. Meanwhile, if you are using different tools for these, then the chances of data leakage while attempting to integrate them also increase. You may also fail to draw any significant insights while wandering directionless through data and drown under them.
Challenge #5: Maintaining a Testing Culture
One of the most crucial characteristics of optimization programs like CRO and A/B testing is that it is an iterative process. This is also one of the major obstacles that businesses and marketers face. For your optimization efforts to be fruitful in the long run, they should form a cycle that roughly starts with research and ends in research.
This challenge is not just a matter of putting in efforts or about having the required knowledge. Sometimes due to resource crunch, businesses rarely or intermittently use A/B testing and fail to develop a proper testing culture.
Challenge #6: Changing Experiment Settings in the Middle of an A/B Test
When you launch an experiment, you must commit to it completely. Try and not to change your experiment settings, edit or omit your test goals, or play with the design of the control or the variation while the test is running. Moreso, do not try and change the traffic allocations to variations as well because doing so will not only alter the sampling size of your returning visitors but massively skew your test results as well.
So, given all these challenges, is A/B testing worth undertaking?
From all the evidence and data available on A/B testing, even after these challenges, A/B testing generates great ROI. From a marketing perspective, A/B testing takes the guesswork out of the optimization process. Strategic marketing decisions become data-driven, making it easier to craft an ideal marketing strategy for a website with well-defined ends. Without an A/B testing program, your marketing team will simply test elements at random or based on gut feeling and preferences. Such data-less testing is bound to fail.
If you start strong with a good website and visitor data analysis, the first three challenges can be easily solved. With the extensive website and visitor data at your disposal, you can prioritize your backlog, and you won’t even have to decide on what to test. The data will do all the talking. With such quality data coupled with your business expertise, formulating a working hypothesis becomes just a matter of going through the available data and deciding what changes will be best for your end goal. To overcome the third challenge, you can calculate the apt sample size for your testing campaign with the help of many tools available today.
The last two challenges are related to how you approach A/B testing. If you treat A/B testing like an iterative process, half of the fourth challenge may not even be on your plate. And the other half can be solved by hiring experts in the field or getting trained on analyzing research data and results correctly. The right approach to tackle the last challenge is to channel your resources on the most business-critical elements and planning your testing program in a way that, with the limited resource, you can build a testing culture.
How To Make an A/B Testing Calendar – Plan & Prioritize
A/B testing should never be considered an isolated optimization exercise. It is always supposed to be part of a wider holistic CRO program and should be treated. An effective optimization program will always have two parts, namely, plan and prioritize. Waking up one day and deciding to test your website is not how things are done in CRO. A good amount of brainstorming, along with real-time visitor data, is the only way to go about it.
In plain words, you begin by analyzing existing website data and gathered visitor behavior data, then move on to preparing a backlog of action items based on them, prioritizing, testing, and then drawing insights for the future. Eventually, when, as marketers, you conduct enough ad-hoc based tests, you would want to scale your A/B testing program to make it more structured. The first step to doing this is making an A/B testing calendar. A good testing calendar or a good CRO program will take you through 4 stages:
Stage 1: Measure
This stage is the planning stage of your A/B testing program. It includes measuring your website’s performance in terms of how visitors are reacting to it. In this stage, you should be able to figure out what is happening on your website, why it is happening, and how visitors are reacting to it. Everything that goes on in your website should correspond to your business goals. So before everything else, you need to be sure what your business goal/s is (are). Tools like Google Analytics can help you measure your goals. Once you have clearly defined goals, set up GA for your website and define your key performance indicators.
Let’s take an online mobile phone cover store as an example. The business goal for this store is to increase revenue by increasing online orders and sales. The KPI set to track this goal would then be the number of phone covers sold.
This stage, however, does not simply end with defining website goals and KPIs. It also includes understanding your visitors. We have already discussed the various tools that can be used to gather visitor behavior data. Once data is collected, log in observations and start planning your campaign from there. Better data means higher sales.
Once the business goals are defined, KPIs set, and website data and visitor behavior data analyzed, it is time to prepare a backlog.
Backlog: “an accumulation of tasks unperformed or materials not processed.” 
Your backlog should be an exhaustive list of all the website elements you decide to test based on the data you analyzed. With a data-backed backlog ready, the next step is formulating a hypothesis for each backlog item. With the data gathered in this stage and its analysis, you will now have enough context of what happens on your website and why. Formulate a hypothesis based on them. For example, after analyzing the data gathered using quantitative and qualitative research tools in the 1st stage, you come to the conclusion that is not having multiple payment options led to maximum prospect customers dropping off on the checkout page. So you hypothesize that “adding multiple payment options will help reduce drop off on checkout page.”
In short, by the end of this stage, you will know the whats and whys of your website.
Stage 2: Prioritize
The next stage involves prioritizing your test opportunities. Prioritizing helps you scientifically sort the multiple hypotheses. By now, you should be fully equipped with website data, visitor data and be clear on your goals. With the backlog, you prepared in the first stage and the hypothesis ready for each candidate, you are halfway there on your optimization roadmap. Now comes the main task of this stage: prioritizing.
In stage 2, you should be fully equipped to identify problem areas of your website and leaks in your funnel. But not every action area has equal business potential. So it becomes imperative to weigh out your backlog candidates before picking the ones you want to test. You must keep in mind a few things while prioritizing items for your test campaign like the potential for improvement, page value and cost, the importance of the page from a business perspective, traffic on the page, and so on.
But how can you ensure that no subjectivity finds its way in your prioritization framework? Can you be 100% objective at all times? As humans, we give loads of importance to gut feelings, personal opinions, ideas, and values because these are the things that help us in our everyday lives. But CRO is not everyday life. It is a scientific process that needs you to be objective and make sound data-backed decisions and choices. The best way to weed out these subjectivities is by adopting a prioritization framework.
There are many prioritization frameworks that even experts employ to make sense of their huge backlogs. On this pillar page, you will learn about the most popular frameworks that most marketers use – the CIE prioritization framework, the PIE prioritization framework, and the LIFT Model.
1. CIE Prioritization Framework
In the CIE framework, there are three parameters on which you must rate your test on a scale of 1 to 5:
- Confidence: On a scale of 1 to 5, 1 refers to a low confidence level, 5 refers to a high confidence level. Select how confident you are about achieving the expected improvement through the hypothesis.
- Importance: On a scale of 1 to 5, 1 refers to low importance, and 5 refers to high importance. Select how crucial the test (for which the hypothesis is created) is.
- Ease: On a scale of 1 to 5, 1 refers to a high difficulty level, whereas 5 refers to a low difficulty level. Rate how difficult it will be to implement the changes identified for the test.
Before you rate your hypotheses, consider these 3 things:
How confident are you of achieving the uplift?
Prototyping the user persona you are targeting can help you determine the potential of a hypothesis. With a sound understanding of your audience, you can make an educated assumption on whether the hypothesis will address the users’ apprehensions and doubts and nudge them to convert or not.
How valuable is the traffic you are running this test for?
Your website may be attracting visitors in large numbers, but not all visitors become buyers. Not all convert. For example, a hypothesis built around the checkout page holds a higher importance than the one built around the product features page. This is because visitors on the checkout page are way deep in your conversion funnel and have a higher chance of converting than visitors on your product features page.
How easy is it to implement this test?
Next comes determining the ease of implementing your test. Try to answer some questions: Would it need a lot of strategizing on your part to implement the hypothesis? What is the effort required to design and develop the solution proposed by the hypothesis? Can the changes suggested in the hypothesis be implemented using just the Visual Editor, or does it warrant adding custom code? It is only after you have answered all these and other such questions should you rate your backlog candidate on the easing parameter.
2. PIE Prioritization Framework
The PIE framework was developed to answer the question, “Where should I test first?”. The whole aim of the prioritization stage in your A/B testing journey is to find the answer to this very question. The PIE framework talks about three criteria that you should consider while choosing what to test when: potential, importance, and ease.
Potential refers to a page’s ability to improve. Meanwhile, the planning stage equips you with all the data you need to determine this.
Importance refers to a page’s value: how much traffic comes to the page. If you have identified a problem page, but there is no traffic on that page, then that page is of less importance when compared to other pages with higher traffic.
The third and final criteria is ease. Ease defines how difficult it is to run a test on a particular page or element. One way to determine the ease of testing a page is using tools like landing page analyzer to determine the current state of your landing pages, estimate the number and scale of change it would require, and prioritize which ones to do or whether to do it at all. This is important from the perspective of resources. Many businesses drop the idea of undertaking an A/B testing campaign because of the lack of resources. These resources are of 2 kinds:
- Human Resource: Even though CRO and A/B testing were used in the past, it is only in the past few years that they have gained a front stage. Because of this, a large segment of the market does not have a dedicated optimization team, and when they do, it is usually limited to a handful of people. This is where a planned optimization calendar comes in handy. With a properly planned and prioritized backlog, a small CRO team can focus its limited resources on high stake items.
- Tools: As popular as CRO and A/B testing are getting, so are hundreds of A/B testing tools- both low end and high. Without a perspective of an expert, if businesses were to pick one out of the lot, say the cheapest one, and start A/B testing every single item on the backlog, they will reach no statistically significant conclusion. There are 2 reasons for this: one, testing without prioritization is bound to fail and not reap any business profits. Two, not all tools are of the same quality. Some tools may be costlier, but they are either integrated with good qualitative and quantitative research tools or are brilliant standalone tools making them more than capable of producing statistically significant. While the other lot may be cheaper and lure businesses during capital crunch and with a huge backlog, but these tools will only be an investment loss to them without any benefits. Prioritization will help you make sense of your backlog and dedicate whatever little resources you have to a profitable testing candidate.
Backlog candidates should be marked on how hard they are to test based on technical and economical ease. You can quantify each potential candidate as business opportunities based on the above criteria and choose the highest scorer. For example, like an eCommerce business, you may want to test your homepage, product page, checkout page, and thank you (rating) page. Now according to the PIE framework, you line these up and mark them potential, importance and ease:
*marked out of a total of 10 points per criteria.
3. LIFT Model
The LIFT Model is another popular conversion optimization framework that helps you analyze web and mobile experiences and develop good A/B test hypotheses. It draws on the 6 conversion factors to evaluate experiences from the perspective of your page visitor: Value Proposition, Clarity, Relevance, Distraction, Urgency, and Anxiety.
With prioritization, you can have your A/B testing calendar ready for execution for at least 6 to 12 months. This will not only give you time and a heads-up to prepare for the test but also plan around your resources.
Stage 3: A/B Test
The third and most crucial stage is the testing stage. After the prioritization stage, you will have all the required data and a prioritized backlog. Once you have formulated hypotheses that align with your goal and prioritized them, create variations and flag off the test. While your test is running, make sure it meets every requirement to produce statistically significant results before closure, like testing on accurate traffic, not testing too many elements together, testing for the correct amount of duration, and so on.
Stage 4: Repeat
This stage is all about learning from your past and current test and applying them in future tests. Once your test runs for the stipulated amount of time, stop the test and start analyzing the data thus gathered. The first thing you will realize is one of the many versions that were being tested had performed better than all others and won. It’s time for you and your team to now figure out why that happened. There can be 3 outcomes of your test:
- Your variation or one of your variations will have won with statistical significance.
- Your control was the better version and won over the variation/s.
- Your test failed and produced insignificant results. Determine the significance of your test results with the help of tools like the A/B test significance calculator.
In the first two scenarios, do not stop testing just because you have a winner. Make improvements to that version and keep testing. In the third scenario, recall all the steps and identify where you went wrong in the process, and re-do the test after rectifying the mistake.
Here is a downloadable A/B testing calendar sample for your reference. To use this spreadsheet, click on the ‘File’ option in the main menu and then click on ‘Make a copy.’
File > Make a copy
When scaling your A/B testing program, keep in mind the following points:
- Revisiting previously concluded test: With a prioritized calendar in place, your optimization team will have a clear vision of what they will test next and which test needs to be run when. Once you have tested each element or most elements in the backlog, revisit each successful as well as failed campaigns. Analyze the test results and determine whether there is enough data to justify running another version of the test. If there is, then rerun the test – with necessary edits and modifications.
- Increasing testing frequency: While you should always be cautious of testing too many elements together, increasing your testing frequency is essential in scaling your testing program. Your optimization team will have to plan it in such a way that none of the tests affect others or your website’s performance. One way to do this is by running tests simultaneously on different web pages of your website or testing elements of the same web page at different time periods. This will not only increase your testing frequency but also, none of the tests will affect others. For instance, you can simultaneously test one element each of your homepage, checkout page, and sign-up page at one time and other elements (1 element at a time) of these pages after the current test concludes.
- Spacing out your test: This flows from the previous point. If you look at the calendar above, you will see that not more than two tests overlap each other at any given week. In a quest to increase your testing frequency, do not compromise with your website’s overall conversion rate. If you have two or more critical elements to be tested on the same web page, space the two out. As pointed earlier, testing too many elements of a web page together makes it difficult to pinpoint which element influenced the success or failure of the test most. Let’s say, for example, you want to test one of your ad’s landing pages. You lock in on testing the CTA to increase sign-ups and banners to decrease the bounce rate and increase time spent. For the CTA, based on your data, you decide to change the copy. For the banner, you decide to test a video against a static image. You deploy both tests simultaneously, and at the conclusion, both your goals were met. The problem here is that data showed that while sign-ups increased from the new CTA, the video (apart from reducing the bounce rate and increasing average time spent on the page) also helped in this. Most of the people who watched the video also ended up signing up. The problem now is that, because you did not space the two tests, it became impossible to tell which element contributed most to the sign-up increase. Had you timed the two tests better, much more significant insights could have been gathered?
- Tracking multiple metrics: You usually measure an A/B test’s performance based on a single conversion goal and put all your trust in that goal to help you find the winning variation. But sometimes, the winning variation affects other website goals as well. The example above is applicable here too. In addition to reducing bounce rate and increasing time spent, the video also contributed to increased sign-ups. To scale your A/B testing program, track multiple metrics so that you can draw more benefits with less effort.
A/B Testing and SEO
As far as SEO implications on A/B testing are concerned, Google has cleared the air on their blog post titled “Website Testing And Google Search.” The important bits from that post are summarized below:
Cloaking – showing one set of content to humans and a different set to Googlebot – is against our Webmaster Guidelines, whether you’re running a test or not. Ensure you’re not deciding whether to serve the test or which content variant to serve, based on user-agent. An example of this would always be serving the original content when you see the user-agent “Googlebot.” Remember that infringing our Guidelines can get your site demoted or even removed from Google search results – probably not the desired outcome of your test.
Only Use 302 Redirects
Run Experiments for the Appropriate Duration
The amount of time required for a reliable test will vary depending on factors like your conversion rates and how much traffic your website gets. A good testing tool should tell you when you’ve gathered enough data to be able to draw reliable conclusions. Once you have concluded the test, you should update your site with the desired variation(s) and remove all elements of the test as soon as possible, such as alternate URLs or testing scripts and markup.
Use rel=”canonical” Links
Google suggests using rel=“canonical” link attribute on all alternate URLs for you to be able to highlight that the original URL is actually the preferred one. This suggestion stems from the fact that rel=“canonical” more closely matches your intent in this situation when compared to other methods like no index meta tag. For instance, if you are testing variations of your product page, you don’t want search engines not to index your product page. You just want them to understand that all the test URLs are close duplicates or variations on the original URL and should be grouped together, with the original URL as the hero. Sometimes, in these instances, using no index rather than rel=“canonical” in such a situation can sometimes have unexpected bad effects.
A/B Testing Examples
A/B testing in Media & Publishing Industry
Some goals of a media and publishing business may be to increase readership and audience, increase subscriptions, increase time spent on their website by visitors, boost video views and other content pieces with social sharing, and so on. You may try testing variations of email sign-up modals, recommended content, social sharing buttons, highlighting subscription offers, and other promotional options.
Any of us who is a Netflix user can vouch for their streaming experience. But not everyone knows how they manage to make it so good. Here’s how – Netflix follows a structured and rigorous A/B testing program to deliver what other businesses struggle to deliver even today despite many efforts – a great user experience. Every change that Netflix makes to its website goes through an intense A/B testing process before getting deployed. One example to show how they do it is the use of personalization.
Netflix uses personalization extensively for its homepage. Based on each user’s profile, Netflix personalizes the homepage to provide the best user experience to each user. They decide how many rows go on the homepage and which shows/movies go into the rows based on the users streaming history and preferences.
They follow the same exercise with media title pages as well. Within these pages, Netflix personalizes what titles we are most likely to watch, the thumbnails we see on them, what title text entices us to click, or if social proof helps make our decision easier, and so on. And this is just the tip of the iceberg.
A/B Testing in eCommerce Industry
Through A/B testing, online stores can increase the average order value, optimize their checkout funnel, reduce cart abandonment rate, and so on. You may try testing: the way shipping cost is displayed and where, if, and how the free shipping feature is highlighted, text and color tweaks on the payment page or checkout page, the visibility of reviews or ratings, etc.
In the eCommerce industry, Amazon is at the forefront in conversion optimization partly due to its scale at and partly due to their immense dedication to providing the best customer experience. Amongst the many revolutionary practices they brought to the eCommerce industry, the most prolific one has been their ‘1-Click Ordering’. Introduced in the late 1990s after much testing and analysis, 1-Click Ordering lets users make purchases without having to use the shopping cart at all.
Once users enter their default billing card details and shipping address, all they need to do is click on the button and wait for the ordered products to get delivered. Users don’t have to enter their billing and shipping details again while placing any orders. With the 1-Click Ordering, it became impossible for users to ignore the ease of purchase and go to another store. This change had such a huge business impact that Amazon patented (now expired) in 1999. In fact, in 2000, even Apple bought a license for the same to use in their online store.
People working to optimize Amazon’s website do not have sudden ‘Eureka’ moments for every change they make. It is through continuous and structured A/B testing that Amazon delivers the kind of user experience that it does. Every change on the website is first tested on their audience and then deployed. If you were to notice Amazon’s purchase funnel, you would realize that even though the funnel more or less replicates other websites’ purchase funnels, every element in it is fully optimized and matches the audience’s expectations.
Every page, starting from the homepage to the payment page, only contains the essential details and leads to the exact next step required to push the users further into the conversion funnel. Additionally, using extensive user insights and website data, each step is simplified to their maximum possible potential to match their users’ expectations.
Take their omnipresent shopping cart, for example.
There is a small cart icon at the top right of Amazon’s homepage that stays visible no matter which page of the website you are on.
The icon is not just a shortcut to the cart or reminder for added products. In its current version, it offers 5 options:
- Continue shopping (if there are no products added to the cart)
- Learn about today’s deals (if there are no products added to the cart)
- Wish List (if there are no products added to the cart
- Proceed to checkout (when there are products in the cart)
- Sign in to turn on 1-Click Checkout (when there are products in the cart)
With one click on the tiny icon offering so many options, the user’s cognitive load is reduced, and they have a great user experience. As visible in the above screenshot, the same cart page also suggests similar products so that customers can navigate back into the website and continue shopping. All this is achieved with one weapon: A/B Testing.
A/B Testing in Travel Industry
Increase the number of successful bookings on your website or mobile app, your revenue from ancillary purchases, and much more through A/B testing. You may try testing your home page search modals, search results page, ancillary product presentation, your checkout progress bar, and so on.
In the travel industry, Booking.com easily surpasses all other eCommerce businesses when it comes to using A/B testing for their optimization needs. They test like it’s nobody’s business. From the day of its inception, Booking.com has treated A/B testing as the treadmill that introduces a flywheel effect for revenue. The scale at which Booking.com A/B tests is unmatched, especially when it comes to testing their copy.
While you are reading this, nearly 1000 A/B tests are running on Booking.com’s website. Even though Booking.com has been A/B testing for more than a decade now, they still think there is more that they can do to improve the user experience. And this is what makes Booking.com the ace in the game. Since the company started, Booking.com incorporated A/B testing into its everyday work process. They have increased their testing velocity to its current rate by eliminating HiPPOs and prioritizing data before anything else. And to increase the testing velocity even more, all of Booking.com’s employees were allowed to run tests on ideas they thought could help grow the business.
This example will demonstrate the lengths to which Booking.com can go to optimize their users’ interaction with the website. Booking.com decided to broaden its reach in 2017 by offering rental properties for vacations alongside hotels. This led to Booking.com partnering with Outbrain, a native advertising platform, to help grow their global property owner registration.
Within the first few days of the launch, the team at Booking.com realized that even though many property owners completed the first sign-up step, they got stuck in the next steps. At this time, pages built for the paid search of their native campaigns were used for the sign-up process.
Both the teams decided to work together and created three versions of the landing page copy for Booking.com. Additional details like social proof, awards, recognitions, user rewards, etc., were added to the variations.
The test ran for two weeks and produced a 25% uplift in owner registration. The test results also showed a significant decrease in the cost of each registration.
A/B Testing in B2B/SaaS Industry
Generate high-quality leads for your sales team, increase the number of free trial requests, attract your target buyers, and perform other such actions by testing and polishing important elements of your demand generation engine. To get to these goals, marketing teams put up the most relevant content on their website, send out ads to prospect buyers, conduct webinars, put up special sales, and much more. But all their effort would go to waste if the landing page that clients are directed to is not fully optimized to give the best user experience.
The aim of SaaS A/B testing is to provide the best user experience and to improve conversions. You can try testing your lead form components, free trial sign-up flow, homepage messaging, CTA text, social proof on the home page, and so on.
Their website homepage and Contact Us page are the most important pages in their funnel. The team at POSist wanted to reduce drop-off on these pages. To achieve this, the team created two variations of the homepage as well as two variations of the Contact Us page to be tested. Let’s take a look at the changes made to the homepage. This is what the control looked like:
The team at POSist hypothesized that adding more relevant and conversion-focused content to the website will improve user experience and generate higher conversions. So they created two variations to be tested against the control. This is what the variations looked like:
Control was first tested against Variation 1, and the winner was Variation 1. To further improve the page, variation 1 was tested against variation 2, wherein variation 2 turned as the winner. The new variation increased page visits by about 5%.
After reading this comprehensive piece on A/B testing, you should now be fully equipped to plan your own optimization roadmap. Follow each step involved diligently and be wary of all major and minor mistakes you can commit if you do not give data the importance it deserves. A/B testing is invaluable when it comes to improving your website’s conversion rates.
If done with complete dedication and with the knowledge you now have, A/B testing can reduce many risks involved when undertaking an optimization program. It will also significantly improve your website’s UX by eliminating all weak links and finding the most optimized version of your website.
If you found this guide useful, spread the word and help fellow marketers A/B test without falling for the most common pitfalls. Happy testing!
Frequently Asked Questions on A/B Testing
A/B testing is the process of comparing two variations of a page element, usually by testing users’ response to variant A vs. variant B and concluding which of the two variants is more effective.
In digital marketing, A/B testing is the process of showing two versions of the same web page to different segments of website visitors at the same time and then comparing which version improves website conversions.
There are various reasons why we do A/B testing. A few of them include solving visitor pain points, increasing website conversions or leads, and decreasing the bounce rate. Please read our guide to know the rest of the reasons.
In A/B testing, traffic is split amongst two or more completely different versions of a webpage. In multivariate testing, multiple combinations of a few key elements of a page are tested against each other to figure out which combination works best for the goal of the test.
Download this Guide
Thanks! We've mailed you the guide, please check your inbox.
Thank you for your time.