How to Create a Multivariate Test

Last updated

Multivariate Testing (MVT) is a method wherein you make changes to few key sections on a webpage and create variations for all the possible combinations that result out of those changes. To learn more on MVT, click here.

The end-to-end process of creating a Multivariate test involves the following steps:

  1. Setting up pages for the Multivariate test
  2. Creating variations using Visual Editor
  3. Adding conversion goals and estimating test duration
  4. Finalizing the test
  5. Previewing and starting the test

To get started with a Multivariate test, go to Multivariate under Test on the navigation menu. This takes you to the Multivariate Tests dashboard that lists down all the Multivariate tests created in the account. To create a new new test, click Create in the upper-right corner of the Multivariate Tests dashboard.mvt-2

Setting Up Pages for the Multivariate Test

To set up the test, you can start by entering a title by clicking on the pencil icon against New Campaign. Then, select a hypothesis you want to test during the test, specify the page URLs, and define targeting options.

  1. Select an existing hypothesis you want to test. Every test in VWO should be associated with one hypothesis against which the results of the test are measured. If you are just starting out and do not have a hypothesis yet, click Create a New Campaign to define a hypothesis for testing. To learn more about hypothesis, click here.mvt-3
  2. Next, specify the URL of the pages where you want to include in the test. You can use URL patterns and regular expressions to run tests site-wide and on multiple pages. To learn more about URL options, click here.
    ATTENTION Please ensure that you have defined the correct webpage URLs for the test. You can make changes only to the pages defined here and test settings are applied only to the relevant pages when rendering the page to the visitors.
     mvt-4
    PRO-TIP Click the Advanced tab next to the URL field to include or exclude URLs from the test. Using this tab, you can also define advanced pattern-matching options to include either a broad URL with wildcard entries or a specific URL.
  3. Under Advanced tab, enter a base URL for the split test views in Enter URL of the page to create previews. This primary or base URL may not be part of a test, but it indicates that the webpage VWO will be used as the primary reference URL for generating test previews and default heatmaps. To learn more about base URL, click here.mvt-5
  4. More Options allows you to configure the following advanced settings for the test: mvt-6
      • Load campaign pages on a mobile device: Enable the option if you have a mobile or a responsive website and want to load VWO Editor in a simulated mobile view. To learn more, click here.
      • Enable campaign for a specific visitor group: Enable the option if you want to target specific visitor segments. You can target these visitors per different metrics such as geographical location, user device, and other user behaviors. To learn more, click here.
  5. Click Next to start creating variations for your test.

Creating Variations Using Visual Editor

VWO Visual Editor allows you to quickly make changes to the existing webpages and create variations to test. In the Variations menu, Control will always be your original webpage while Variation 1 will be the modified version. You can add more variations to test a different set of changes. You can switch between the Control and the Variation from the Variations tab on the editor.ab-6

To make change to your the webpage, click on the element that you want to edit. It is always recommended to test changes on important elements such as the headline or a Call-to-Action button (CTA). Try a major change instead of small ones so that you can get and observe a significant change in the result.

Edit Tools

You can select the kind of edits you want to make. You can edit text; change colors of elements (such as buttons); move, rearrange, hide, or delete elements; or make changes directly to the HTML code snippet. You can also edit images by cropping them, changing contrast, and so on. To read more about various edit tools, click here.

By default, your changes are made directly to a variation of your page. Control, by the virtue of being the original page, cannot be changed. To create another variation of your page with a different set of modifications to the test, click Add JavaScript/CSS Section.mvt-7

PRO-TIP

It is always recommended to test changes on important elements such as a headline or a call-to-action (CTA) button. Try a major change instead of minor ones so that you can observe a significant change in the result. To learn more, read our best-practices guide on testing.

[https://vwo.com/knowledge/vwo-editor-best-practices]

Setting up goals using Visual Editor

In addition to creating variations, Visual Editor can also be used to define three types of goals—tracking clicks on a link, tracking clicks on an element, and tracking form submits. To learn more on how to create goals using Visual Editor, click here.ab-7You can add multiple goals for the same test using Visual Editor. All the goals you have added using the editor will be listed on the next screen where you must indicate one goal from the list as a primary goal.

Use URL Patterns to track clicks on multiple links together. For instance, if you have many blog articles on your test page and want to track how many users clicked the link to any blog article, then you can set up the goal to track clicks to the URL Pattern ‘/blog/*’. To learn how to use URL patterns, click here.

Clicking Next will take you to the Conversion Goals page where you can review the goals you have added through the editor or add new goals to track other user activities.

Adding Conversion Goals

Conversion goals are user activities you want to track on your website. A conversion goal can be clicks on buttons and links, form submits, or just an engagement among others. Goals allow you to compare the performance of control against variations by collecting data to measure the impact on your conversion rate or sales revenue. You can track multiple goals in a single test.mvt-8

  1. Select the visitor activity you want to track for actions such as page visits, clicks on links, elements, engagement on page, and so on. For example, if you have modified a CTA button on a page, you may want to track how many visitors landed on the page and how many clicked the button.
  2. Select a URL type using the dropdown and enter URL details for the pages where you want to track the visitor activity. To learn more about different types of conversion goals, click here.
  3. Click Include URL to add more pages to the test.
  4. Click Add Another Goal to add another activity you want to track.
  5. Click Next for the Duration Calculator screen.

Estimated Duration of the Test

On this page, you can enter the existing data of your test pages to configure VWO SmartStats for the best possible results. Using your current data and the expected conversion lift you want, VWO SmarStats will also give you an estimate of how long you’ll need to run the test.

To know the details and obtain data regarding your control page, you can refer to any existing analytics tool you are using (Google Analytics, Mixpanel, Kissmetrics, and so on) or run a Conversion Tracking test in VWO. To learn more about VWO Smartstats, click here.mvt-9

The Duration Calculator screen calculates the estimated duration through the following:

    • Estimated Existing Conversion Rate: This is the conversion rate of the control version of your website. A tool like Google Analytics or a VWO Conversion Tracking test can help you measure this rate.
    • Minimum improvement in conversion rate you want to detect: This is the percentage of improvement in conversion you want to achieve against the existing conversion rate. If the variation actually achieves this level of lift over control, then the test has a 90% chance of correctly reporting variation a winner.
    • Number of combinations (including control): This includes all the different variations you are running the test on, plus the Control.
    • Average number of monthly visitors: Check your analytics tool or run a VWO conversion test to find out the number of monthly visitors.

Advanced options

    • p-value cut-off: The probability of falsely reporting a winner even when all variations have identical conversion rates.
    • Statistical power: The probability of detecting a change at least as large as the desired lift. For example, if the desired lift is 10% and statistical power is 80%, then if a variation with 10% lift is present, your test has an 80% chance of detecting it.
ATTENTION The estimated duration may differ from the initial estimate in the duration calculator if the actual number of visitors differs significantly from what you entered.

Click Next to finalize your test.

Finalizing the Test

In the final step of your test creation, you can add notes and labels to your test. You can allocate traffic percentage for the Multivariate test, enable integration with third-party products such as Google Analytics, and set up other advanced options that give you more control over the test.mvt-10

Allocate Traffic to Your Test

Specify the percentage of website visitors you want to include in your test. Click the percentage figure to change the traffic volume or move the slider to increase or decrease traffic allocation.

PRO-TIP If you get a lot of traffic to the page you are testing, then you could lower the percentage, but it might take longer to get a concrete result.

Enable Heatmaps

Enable this option if you want to generate a heatmap report for the test. By default, the option is enable for all tests. To learn more about heatmaps, click here.

Integrate VWO with Third-Party Products

mvt-11

VWO also lets you push your test data to an external tool. You can integrate VWO with a suite of third-party products to access the VWO data on your preferred tool. To learn more about integrating VWO with third-party products, click here.

Exploring Advanced Options

You can personalize your test even further by exploring Advanced Options.mvt-12

Enable Scheduling

This option allows you to schedule your tests to run, pause or stop at specific times. To learn how to schedule a test, click here.

Enable test for multiple domains

Select this option if your test involves multiple domains and you want to track conversions across the domains. Enabling this option allows VWO to set a cookie that tracks the variation a user has seen, goals completed, and user movement across multiple domains.

To learn more about cross-domain tracking, click here.

Hide test names in visitor settings

Use this option to replace your actual test or variation information with a standard content format when visitors try to view the information using the “view page source” option in their browsers.

For example, if you have a test named “Personalized test for Facebook users,” VWO will replace it as “Test-numerical ID.” If this option is enabled, website visitors cannot view your sensitive details, such as test name and variations name.

Please note that you cannot enable this option for the tests that are already running, paused or stopped.

Host JavaScript files on your server

Select this option to host your JavaScript files on your own server. Self-hosting JavaScript files on your server eliminates the need to depend on VWO servers to render the files required for your test.

If you enable the self-hosting option, VWO automatically disables multi-domain support for that test.

Enable report segmentation

Select this option to generate reports, based on different visitor segments. You can filter and view the collected reports data for various visitor segments to determine which test works for specific segments of your website visitors.

Report segmentation is a premium feature and is bundled with the VWO Enterprise plan only.

Preview and Start Test

mvt-13

To run the test, you need to add the VWO Smart Code to the webpages you want to test. We recommend that you put the code snippet on all pages of your website. Copy the following code snippet and send it to your Developer or you can add it to the webpages yourself. Please make sure to enter your account_id in the code. To learn how to add VWO Smart Code to your website, click here.

<!– Start Visual Website Optimizer Asynchronous Code –>
<script type='text/javascript'>
var _vwo_code=(function(){
var account_id=12345,
settings_tolerance=2000,
library_tolerance=2500,
use_existing_jquery=false,
/* DO NOT EDIT BELOW THIS LINE */
f=false,d=document;return{use_existing_jquery:function(){return use_existing_jquery;},library_tolerance:function(){return library_tolerance;},finish:function(){if(!f){f=true;var a=d.getElementById('_vis_opt_path_hides');if(a)a.parentNode.removeChild(a);}},finished:function(){return f;},load:function(a){var b=d.createElement('script');b.src=a;b.type='text/javascript';b.innerText;b.onerror=function(){_vwo_code.finish();};d.getElementsByTagName('head')[0].appendChild(b);},init:function(){settings_timer=setTimeout('_vwo_code.finish()',settings_tolerance);var a=d.createElement('style'),b='body{opacity:0 !important;filter:alpha(opacity=0) !important;background:none !important;}',h=d.getElementsByTagName('head')[0];a.setAttribute('id','_vis_opt_path_hides');a.setAttribute('type','text/css');if(a.styleSheet)a.styleSheet.cssText=b;else a.appendChild(d.createTextNode(b));h.appendChild(a);this.load('//dev.visualwebsiteoptimizer.com/j.php?a='+account_id+'&u='+encodeURIComponent(d.URL)+'&r='+Math.random());return settings_timer;}};}());_vwo_settings_timer=_vwo_code.init();
</script>
<!– End Visual Website Optimizer Asynchronous Code –>

Congratulations! You’ve successfully created your Multivariate Test.

Contact Us / Login

Product
Resources Home