How to Create an A/B Test

Last updated

A/B testing involves creating two or more versions of a webpage or app and comparing these to determine which version performs better. To learn more about A/B testing, click here.

To get started with A/B testing, go to A/B under Test on the navigation menu. This will take you to the A/B test dashboard where you can view the lists of all A/B tests created in the account. To create a new test, click Create in the upper right-hand corner of the A/B Tests dashboard.

ab-1

The end-to-end process of creating A/B tests involves the following steps:

  • Setting up pages to A/B test
  • Creating variations using Visual Editor
  • Adding conversion goals and estimating test duration
  • Configuring advanced options
  • Previewing and starting a test

Setting Up Pages to A/B Test

To set up a test, you can start by entering a name for the test. Then, select a hypothesis you want to test, enter URLs of the pages, and define targeting options. To assign a name to the test, click the pencil icon against New Test.

ab-2

Select Hypothesis to Test

Select an existing hypothesis you want to test. Every test in VWO should be associated with one hypothesis against which the results of the test are measured. If you are just starting out and do not have a hypothesis yet, click Create a New Hypothesis to define a hypothesis. To learn more about hypothesis, click here.

ab-3

Enter URLs to Test

Next, specify the URL of the pages  you want to include in the test. You can use URL patterns and regular expressions to run site-wide tests. To learn more about URL options, click here.

ATTENTION Please ensure that you have defined the correct webpage URLs for the test. You can make changes only to the pages defined here, and test settings are applied only to the relevant pages when rendering the page to the visitors.

ab-4

Click Advanced next to the URL field to include or exclude URLs from the test. Using this tab, you can also define advanced pattern-matching options to include either a broad URL with wildcard entries or a specific URL. To learn more about URL options, click here.

Configure More Options

More Options allows you to configure the following advanced settings for the test:

ab-5

  • Open in Code-only Mode: This is an advanced mode for developers that lets you create variations in a code-only mode instead of using the Visual Editor. When you select this mode and click Next, Code Editor opens. Variations created using this mode cannot be further edited using Visual Editor. To learn more about Code Editor, click here.
    ATTENTION The Code-only mode only supports custom JavaScript. You can add any relevant code for firing the event such as DOM Ready and window.load.
  • Load test pages on a mobile device: If you are creating your test for a mobile device, select this option will to load the VWO Editor in a simulated mobile view instead of the default desktop view. To learn more about creating A/B tests for mobile websites, click here.
  • Enable test for a specific visitor group: You can target your test to a specific visitor segment per different conditions such as geographical location, user device, and other user behaviors. To learn about defining visitor segments , click here.
  • Click Next to start creating variations for your test.

Creating Variations Using Visual Editor

VWO Visual Editor allows you to quickly make changes to the existing webpages and create variations to test. In the Variations menu, Control will always be your original webpage while Variation 1 will be the modified version. You can add more variations to test a different set of changes. You can switch between the Control and the Variation from the Variations tab on the editor. To learn about using Visual Editor, click here.

ab-6

To make change to your the webpage, click on the element that you want to edit. It is always recommended to test changes on important elements such as the headline or a Call-to-Action button (CTA). Try a major change instead of small ones so that you can get and observe a significant change in the result.

Edit Tools

Using the edit tools, you can edit text; change colors of elements (such as buttons); move, rearrange, hide, or delete elements; or make changes directly to the HTML code. You can also edit images by cropping them, changing contrast, and so on. To read more about the various edit tools, click here.

By default, your changes are made directly to the ‘Variation’ of your page. ‘Control’ is always your original page and cannot be changed. Click Add Variation to create another variation of your page with a different set of changes to the page.

Setting Up Goals Using Visual Editor

In addition to creating variations, Visual Editor can also be used to define three types of goals—tracking clicks on a link, tracking clicks on an element, and tracking form submissions. To learn more on how to create goals using Visual Editor, click here.

ab-7

You can add multiple goals for the same test from Visual Editor. All the goals you have added using the editor will be listed on the next page where you must indicate one from the list as a primary goal.

Use URL Patterns to track clicks on multiple links at the same time. For instance, if you have many blog articles on your test page and want to track how many users clicked the link to any blog article, you can set up the goal to track clicks to the URL Pattern ‘/blog/*’.

Clicking Next will take you to the Conversion Goals page where you can review the goals you have added through the editor or add new goals to track other user activities.

Adding Conversion Goals and Estimating Test Duration

Conversion goals are user activities you want to track on your website. A conversion goal can be clicks on buttons and links, form submits, engagement on a page, and so. Goals allow you to compare the performance of Control against variations by collecting data to measure the impact on your conversion rate or sales revenue. You can track multiple goals in a single test.

ab-8

  • Select a visitor activity you want to track for actions such as page visits, clicks on links, elements, engagement on a page, and so on. For example, if you have modified a CTA button on a page, you may want to track how many visitors landed on the page and how many clicked the button.
  • Select a URL type using the drop-down and enter URL details for the pages where you want to track the visitor activity. To learn more about different types of conversion goals, click here.ab-9
  • Click Include URL to add more pages to the test.
  • Click Add Another Goal to add another activity you want to track.

Estimated Duration of the Test

On this page, you can enter the existing data of your test pages to configure VWO SmartStats for the best possible results. Using your current data and the expected conversion lift you want, VWO SmarStats will also give you an estimate of how long you’ll need to run the test.

To find out the data regarding your Control page, you can refer to any existing analytics tool you are using (Google Analytics, Mixpanel, Kissmetrics, and so on) or run a Conversion Tracking test in VWO to get accurate data. To learn more about VWO Smartstats, click here.

ab-10

  • Average number of monthly visitors: Enter the average number of monthly visitors to your webpage over the last few months.
  • Conversion rate for Control: Enter the current conversion rate of the control page. This conversion rate is specific to the goal you are testing for. Entering the conversion rate of your entire website will lead to inaccurate results.
  • Minimum lift in conversion rate you care about: Enter the improvement in the conversion rate you want to achieve through this test. For the increase in conversion rate, you should enter a realistic value. In general, a 15–20% change is worth caring about. For instance, if the current conversion rate of your page is 6%, then you may be interested only if the conversion rate of the variation goes up by 20%. (A 20% increase will boost your conversion rate from 6–7%.)
  • Certainty vs. Speed: Select how you want to configure the test in terms of Certainty versus Speed.
    • Quick learning: Choose this mode for finding quick trends where tests don’t affect your revenue directly. This is the ideal mode when testing non-revenue goals such as the bounce rate and time spent on a page or for quick headline tests. Through this mode, you can reduce your testing time for non-critical tests by deploying a quick winner when there isn’t a risk of hurting your revenue directly.
    • Balanced: As the name suggests, it is the best balance between the testing time and minimizing the potential loss. This mode is used during most of the tests.
    • High certainty: Best for revenue-critical tests when you want to absolutely minimize the potential loss. In general, it takes the longest time to conclude a test. Suppose you have an e-Commerce website and you want to test changes to the checkout flow. You want to be sure enough to minimize the potential loss from deploying a false winner even if it takes a lot of time. This is the best mode for such critical tests that affect your revenue directly.
  • To finalize your test, click Next.

Configuring Advanced Options

You’re almost done! You can now go on to naming your test and adding notes and labels to it. You can decide the percentage of traffic you want to expose your A/B Test to. You can also integrate VWO data with third-party products such as Google Analytics. You can take customization further by accessing Advanced Options that provide you more control over the A/B test.

ab-11

1. Allocating Traffic to Your Test

Specify the percentage of website visitors you want to include in your test. Click the percentage figure to change the current traffic volume or move the slider to increase or decrease traffic allocation.

PRO-TIP If you get a lot of traffic to the page you are testing, then you could lower the percentage, but it might take longer to get a concrete result. Check out our Sample Size Calculator.

2. Enable Heatmaps

Enable this option if you want VWO to capture data to generate a heatmap report for the test. By default, the option is enabled for all tests. To learn more about heatmaps, click here.

Integrate VWO with Third-Party Products

ab-12

VWO also lets you push your test data to a host of external tools. You can integrate VWO with a suite of third-party products to access the VWO data on your preferred tool. To learn more about integrating VWO with third-party products, click here.

Advanced Settings

ab-13

  • Enable Scheduling: This option allows you to schedule your tests to run, pause or stop at specific times. To learn how to schedule a test, click here.
  • Enable test for multiple domains: Select this option if your test involves multiple domains and you want to track conversions across these domains. Enabling this option allows VWO to set a cookie that tracks the variation a user has seen, goals completed, and user movement across multiple domains. To learn more about cross-domain tracking, click here.
  • Customize traffic percentage for each variation: You can split your website traffic among the different variations you created for the test. Splitting visitor traffic across variations ensures that all the variations you created are proportionately tested.
    For example, you may want to split your traffic among three different variations of a test page to find out which variation gets you maximum conversions. To learn more about how to divide traffic amongst each variation, click here.
  • Hide test names in visitor settings: Use this option to replace your actual test or variation information with a standard content format when visitors try to view the information using the View Page Source option in their browsers. Note that you cannot enable this option for the tests that are already running, paused, or stopped.
    For example, if you have a test named Personalized test for Facebook users, VWO will replace it as Test-numerical ID. If this option is enabled, website visitors cannot view your important test details, such as test name and variations name.
  • Host test files on your server: Select this option to host your JavaScript files on your own server. Self-hosting JavaScript files on your server eliminates the need to depend on VWO servers to render the files required for your tests. If you enable the self-hosting option, VWO will automatically disable multi-domain support for that test.
  • Enable report segmentation: Select this option to generate test reports based on different visitor segments. You can filter and view the collected reports data for various visitor segments to determine which test works for specific segments of your website visitors. Report segmentation is a premium feature and is bundled with the VWO Enterprise plan only.

Previewing and Starting Test

To run the test, you need to add the VWO Smart Code to the webpages you want to test. We recommend that you put the code snippet on all the pages of your website. Copy the following code snippet and send it to your Developer, or you can add it to the webpages yourself. Please make sure to enter your account_id in the code. To learn how to add VWO Smart Code to your website, click here.

<!– Start Visual Website Optimizer Asynchronous Code –>
<script type='text/javascript'>
var _vwo_code=(function(){
var account_id=12345,
settings_tolerance=2000,
library_tolerance=2500,
use_existing_jquery=false,
/* DO NOT EDIT BELOW THIS LINE */
f=false,d=document;return{use_existing_jquery:function(){return use_existing_jquery;},library_tolerance:function(){return library_tolerance;},finish:function(){if(!f){f=true;var a=d.getElementById('_vis_opt_path_hides');if(a)a.parentNode.removeChild(a);}},finished:function(){return f;},load:function(a){var b=d.createElement('script');b.src=a;b.type='text/javascript';b.innerText;b.onerror=function(){_vwo_code.finish();};d.getElementsByTagName('head')[0].appendChild(b);},init:function(){settings_timer=setTimeout('_vwo_code.finish()',settings_tolerance);var a=d.createElement('style'),b='body{opacity:0 !important;filter:alpha(opacity=0) !important;background:none !important;}',h=d.getElementsByTagName('head')[0];a.setAttribute('id','_vis_opt_path_hides');a.setAttribute('type','text/css');if(a.styleSheet)a.styleSheet.cssText=b;else a.appendChild(d.createTextNode(b));h.appendChild(a);this.load('//dev.visualwebsiteoptimizer.com/j.php?a='+account_id+'&u='+encodeURIComponent(d.URL)+'&r='+Math.random());return settings_timer;}};}());_vwo_settings_timer=_vwo_code.init();
</script>
<!– End Visual Website Optimizer Asynchronous Code –>

Contact Us / Login

Product
Resources Home