How to Create an A/B Test for a Mobile Application

Last updated

A/B Testing a mobile app means testing two different versions of the mobile application and tracking the respective conversion performance. Using VWO, you can compare and test different calls to action, layouts, and design elements to optimize your mobile app and deliver the best possible product to your users.

ATTENTION Mobile App Testing developer documents are at another location. Please use the following links to access the detailed documentation and FAQs:

The end-to-end process for mobile app A/B testing involves the following steps:

  • Create application for testing
  • Create app variations
  • Preview variation on device
  • Add conversion goals
  • Configure advanced options
  • Start the mobile app A/B test

Create Application for Testing

  1. To create an A/B test for a mobile application, click Mobile A/B under Test and click Create on the upper-right side of the screen.mobile-1
  2. To run a mobile app test, you need to first add your mobile app in VWO. As you add an app, VWO generates a unique API key for the app. The API key is used during SDK setup in the mobile app.mobile-2
  3. In the Add an Application screen, enter details of the mobile application you want to integrate. mobile-3
  4. Enter the name of the application you want to add.
  5. Select a platform supported by your application—iOS or Android. Click to read about the instructions on integrating iOS or Android apps.
  6. VWO mobile SDK is generally integrated in your mobile app code by an engineer. To share installation instructions and api key, click Send To Developer and share details with someone in your engineering team.
  7. To create the variations, click Next.

Create App Variations

The Variation(s) screen has an integrated online editor where you can code the changes you want to implement for the application variations.mobile-4

You can freely write JSON for the variations; it can be nested JSON as well. Note that JSON must start with a curly brace that should be followed by a JSON object.

Preview Variation on Device

Previewing the variation allows you to gauge how the variation will appear on a user’s device. To enable previewing, integrate your app with VWO SDK using the API key and then run it on a device or simulator. Once you run the app, the Device icon is displayed next to Preview.

Click the Device icon and then select the device/simulator you want to preview the variation on.  When the app starts running, Click on PREVIEW ON DEVICE button when you want the mobile app to receive the JSON variation.

As the mobile app successfully receives the variation, the text on the button changes to Variation Sent.

To implement the changes, call variationForKey on the SDK and the changes will be updated. Do not disconnect the browser or mobile app.
Please refer to relevant documentation for iOS preview mode and Android preview mode for details.

 

Add Conversion Goals

Conversion goals help you measure the success of your test. Define goals to track the relevant conversion events for your requirement.mobile-6

You can add the following goal types:

  • Generates Revenue: Tracks revenue from purchases or sales made through your application by defining a value for every successful transaction on your application.
  • Trigger Custom Conversion: Tracks goal conversion on any custom event such as taps, page views, or any other user action. For example, you can define a custom conversions goal to track taps on a pop-up event on your application or to track how many users viewed the pop-up. To learn more about adding Mobile App goals, click here.

Estimated Duration

Use VWO SmartStats to calculate the estimated duration required for the A/B test to present you best results. Enter details about your existing app traffic to get duration estimates per your current traffic and targeted improvement.

To learn more and obtain data regarding your control page, refer to any existing analytics tool you are using (Google Analytics (GA), Mixpanel, Kissmetrics, and so on) or run a Conversion Tracking in VWO.mobile-7

  • Average number of monthly visitors: Type the average of monthly visitors to your mobile app.
  • Conversion rate for Control: Type the current conversion rate of the app. Conversion rate is the the number of people who complete a goal conversion against the total number of visitors.
  • Minimum lift in conversion rate you care about: Type the improvement in the conversion rate you want to achieve through this test. For the increase in the conversion rate, you should choose a realistic value. Usually a 15–20% change is worth caring about. For example, if the current conversion rate of your app is 6%, then you may be interested only if the conversion rate of the variation goes up by 20%. (A 20% increase will boost your conversion rate from 6% to 7%.) To learn more about the calculations, click here.
  • Certainty vs. Speed: Select how you want to configure in terms of Certainty versus Speed.
    • Quick learning: For finding quick trends where tests don’t affect your revenue directly. You can choose this mode for testing non-revenue goals such as the bounce rate and time spent on a page, or for quick headline tests. Through this mode, you can reduce your testing time for non-critical tests where there isn’t a risk of hurting your revenue directly by deploying a false winner.
    • Balanced: Ideal for most tests. This mode is used during most of the tests. As the name suggests, it is the best balance between the testing time and minimizing the potential loss.
    • High certainty: Best for revenue-critical tests when you want to absolutely minimize the potential loss. In general, it takes the longest to conclude a test. Suppose you have an e-Commerce website and you want to test changes to the checkout flow. You want to be sure enough to minimize the potential loss from deploying a false winner even if it takes a lot of time. This is the best mode for such critical tests, which affect your revenue directly.
  • To finalize your test, click Next.

Configure Advanced Options

You’re almost done! You can add name, add notes and labels, and allocate traffic percentage for the test. You can take customization further by accessing Advanced Options that provide you more control over the A/B Test.

1. Allocate traffic percentage

Specify the percentage of app visitors you want to include in your test. Click the percentage figure to change the traffic volume or move the slider to increase or decrease traffic allocation.

If you get a lot of traffic to the page you are testing, you can lower the percentage, but then it might take longer to get a concrete result.

2. More Options

  • Enable campaign for a specific user group: Enable the option to target the test for a specific user group. Using the option, you can group your app users into different segments. To learn how to create visitor segments, click here.
  • Enable scheduling: You can schedule your test to run, pause, or stop at specific times. To learn how to schedule your test, click here.
  • Customize traffic percentage for each variation: You can split your app traffic among the different variations you created for the test. Splitting visitor traffic among variations ensures that all the variations you created are proportionately tested.

For example, you may want to split your traffic among three different variations of a test page to find out which variation gets you the maximum conversion.

Start the Mobile App A/B Test

To start the test, click Start Now. To schedule it for a later time, click Start Later.mobile-9

Contact Us / Login

Product
Resources Home