Defining the Conversion Rate Optimization Process

At the simplest, the conversion rate optimization process involves 5 steps.

  • Research Phase: Identify areas of improvement.
  • Hypothesis Phase - Construct an educated hypothesis.
  • Prioritization Phase - Prioritize test ideas.
  • Testing Phase - Choose the right test and set it up.
  • Learn Phase - How to analyze test results.

Expert optimizers like Michael Aagaard have been doing optimization for a while now and have distilled their learning into frameworks that help you do optimization as a repeatable process. A number of conversion rate optimization frameworks exist to help optimizers plan and proceed with optimization. Right below, you’ll find an excellent resource that rounds up the most popular CRO frameworks we have today.

Research Phase - Identify Areas of Improvement in the Funnel

Only one in every seven A/B tests gives a winning result. But if the test is performed by a CRO specialized agency, the winning result probability rises to 1 in 3. Why? Research.

Often, marketers resort to copying best practices and tests that yielded results for other firms in a bid to quickly optimize their conversion rate. And they fail, because every orange button does not convert and every long form page does not falter.

Agencies fare better at optimization, because it is what they focus on. Consequently, they put in the effort to understand the specific business model of their customers and do tests based on research.

Step 1: Understanding What Users Do

The first step to do is to get familiar with your analytics dashboard. Here’s what Google Analytics can tell you.

What are visitors doing?

With analytics data, you can zero in on pages that are leaking visitors. How’s the exit rate on each of the pages in the funnel? Is the bounce rate on your home page too high? Finding such symptoms on your pages can help you zero in on where to focus your optimization efforts.

How page features and pages of the funnel shape user behavior?

Conversion Funnel for eCommerce Websites

By setting up an event tracking using Google Tag Manager, you can segment your audience and see how different features on a page influence user behavior. For instance, you might find out that users who use a sorting tool on your category page are likelier to become paying customers than users who don’t. Getting such insights help you weed out unwanted features and concentrate on the features that convert users better.

Step 2: Understanding How Users Behave

Visitor Behaviour Analysis Capabilities

For cases where “what users do” doesn’t give a clear idea of areas of improvement, you need to dig deeper to know how users behave. Visual analytics employs tools like heatmaps, visitor recordings, scroll maps, and form analysis to show exactly how users behave on your pages and forms. Studies have shown that mouse movements are strongly correlated with eye movement and can be used to identify where users focus their attention.

Here’s an example. Data analytics can tell you that visitors are exiting your product page at an alarmingly high rate. Visitor recordings go a step further to actually show recorded sessions of your visitors on the product page, helping you visualize where the users spend most of the time on the page, areas they get stuck on, information they don’t seem to locate, and so on.

Step 3: Understanding Why Users Do What They Do

Over-reliance on data isn’t such a good thing either. There are times when data isn’t conclusive—“when,” “what,” and “how” do not provide enough data to know why. For instance, consider you have a high cart abandonment rate and want to change that.

Surveys help you accurately pinpoint the actual reasons why potential customers abandon their shopping carts. If you are struggling to define who your customer is, survey responses can help carve out your customer personas which can then be used to refine your marketing communication.

Collecting qualitative data through On-page Surveys

Quantitative data, while it is exact, cannot capture the very human nature of consumer behavior. It is erratic, at times, even irrational. Surveys, in their qualitative nature, bring out these factors that numbers fail to decode.

When the research is complete, you’ll be able to identify pages which have room for improvement. What you need to do is decide an expected improvement. Use benchmarking studies and analytic data to decide the improvement you can expect through the proposed change. It is important to arrive at a quantified expected conversion rate because that gives your testing efforts a direction. Else, you might end up improving the conversion rate on a page by 1% and sit cozy without realizing its actual potential.

The next step is to build your hypothesis.

Hypothesis Phase - Construct an Educated Hypothesis

Using this point in your research phase, you should have uncovered enough insights to make an educated guess about what changes to your pages or funnel can bring about a desired change.

At its core, a hypothesis is a statement that consists of 3 parts:

You believe.:

  • ..a particular change..
    based on insights gleaned from quantitative and qualitative data
  • ..will have a particular effect..
    the goal; the conversion metric that you want to improve
  • ..due to a particular reason.
    the rationale behind why you believe the change will have the desired effect

Here’s an example of a good hypothesis.

I believe moving trust signals closer to the billing form will result in 5% more checkouts because it instills confidence in the payment gateway.

Based on the hypothesis, you make some changes to the original page. This new page(s) is/are the variation(s). The objective of the test will be to find out which of these two (or more) pages converts visitors better.

A structured hypothesis also implicitly paves the direction for your optimization efforts. Even if your path fails, you can retrace your steps and correct the path. Without this structured process, optimization efforts may go astray and lose their purpose. Here’s what an unstructured, unscientific hypothesis looks like.

“Let’s just try changing that button color because it worked out for companies A, B, and C.”

That’s the kind of hypothesis you don’t want to end up with.

Prioritization Phase - Choose an Order

After you uncover areas of your funnel for optimization, plan out your testing schedule. What do you test first?

Several frameworks exist to help us out here. One of the most popular is the P.I.E. framework formulated by Chris Goward at WiderFunnel:

Potential

Find out the pages that are performing worst and can improve greatly.

Importance

Then narrow down by selecting the ones that have the most valuable traffic. Traffic is valuable when it’s either costly (paid) or super relevant to your product offering.

Ease

Even when you have a final list of pages, it’s important to realize that not all pages are easily optimized. A page, such as an eCommerce product listings page, may be technically complicated to start optimizing while another, such as your home page, may have too many stakeholders to please. It’s important to go for the one that is easily optimized first and then move up the list.

Testing Phase — A/B, Split, or Multivariate?

Before you run a test, there are a couple of things to understand

  • What is statistical significance, and why is it critical?
  • How long do you need to run a test?
  • What should I use—A/B, Split, or Multivariate test?
  • How not to run an optimization test

What Is Statistical Significance and Why Is It Critical?

The reason we run tests is to understand if a particular change(s) can yield better conversions. Let’s say you’ve started a test, and it runs on the first 10 visitors. You see that 2 of those visitors converted on the variation, compared to 1 on the original page. That’s a commendable 20% conversion rate against a paltry 10% on the original.

But does that guarantee a 20% conversion rate, consistently?

Probably not, because these 10 visitors might not be a good representation of maybe the 1,000 visitors that your website gets everyday.

So then, at what visitor volume can you safely say, “Okay, I’ve tested enough to conclude that this page will consistently deliver a better conversion rate.”

Enter statistical significance.

Consider another example to make this as easy to understand as possible.

You’ve run the test. The test results show that the variation outperforms the control with 95% statistical significance.

What this really means is that there is a 5% chance that the variation outperformed the control purely by accident.

Remove “that” after “Or” at both instances. Retain the comma after “Or.” you can be 95% certain that the variation will give the expected result if deployed.

As a thumb rule, a statistical significance of 95% or higher is a good time to stop the test…

..provided you have run the test long enough. In the next section, we cover why test duration is important.

How Long Do You Need to Run an A/B Test for Dependable Results?

You need to decide a test duration before you start running a test.

When you run a test, visitors are constantly included in the test and the numbers keep changing. Conversion rates may rise, dip, and stagnate at different times through the test. As statistical significance is displayed throughout a test, it can show high significance before the test has a chance to run through its intended duration.

So depending on when you choose to see the results of a test, the statistical significance a test displays could be high or low.

This gives rise to the problem of “peeking.”

Peeking error

If you peek at the result of a test before it is run long enough (or collected enough samples), there is a chance that you’ll find statistical significance that is high and that you decide to stop the test, believing the test has run long enough. This can result in you deploying the actually worse version of a page and hurting your conversions.

Therefore, it’s necessary to decide a duration and declare a winner only after the test has run for that duration.

The required test duration is dependent on the number of visitors your website receives and the expected conversion rate you are looking for. You can use this free test duration calculator to find the period for which you should run your tests.

Bayesian vs. Frequentist A/B Testing

Traditional A/B test engines use what is called the Frequentist method to make statistical computation and declare a winner. This is what requires traditional A/B tests to have a test duration (based on a sample size).

But the fact of the matter is, businesses looking to scale up rapidly do not have the luxury of time at their disposal. So an A/B test engine that circumvents the problem of waiting till a test runs for a predetermined duration while still enabling rational business decisions, was necessary.

This thought gave rise to the Bayesian stats engine that now runs behind the VWO A/B testing platform. Bayesian tests can give statistically significant, actionable results almost 50% faster compared to the older Frequentist method. See for yourself—Bayesian vs. Frequentist.

I’ll not get into the details of how Bayesian achieves this. But at the core of it, Bayesian method tries to tell you “at any point, given the data so far, what’s the probability that B has a higher conversion rate than A.” Neither does it have a time limit built into it nor does it require you to have a deep understanding of statistics. All that is taken care of by the Bayesian engine so you can simply focus on bettering your conversion rate.

What Should You Use—A/B, Split, or Multivariate?

Users are often confused about when they should choose one kind of testing over another. Here are some points to note:

  1. A/B, Split, and Multivariate are not 3 alternatives. Each is a method to do different tasks, and the decision to use one should depend entirely on the task at hand.
  2. For reasons of simplicity, consider A/B testing to be the default testing method, used most often when design changes aren’t complex.
  3. Split testing (or split URL testing) is used when:
    • Design requires such heavy modifications to the original page that creating a separate page (housed on a different URL) is easier.

    • Back-end changes are necessary: Testing a pricing page that is linked to tables at the back end.

    • Pages to be tested already exist on different URLs.

  4. Multivariate testing is used when there are multiple changes proposed to a single page, and you want to test each combination of these changes.

How Not to Run an Optimization Test

Till a few years ago, traffic generation was a predominant function of marketing. Mainstay lead generation channels were getting clogged, and growth through these channels was sluggish. Then came along a slew of A/B testing tools, making what was reserved for the statistically sound accessible to even the aspiring marketer. An entire generation of marketers had their first experience of optimization using these tools. Sure enough, A/B testing led to considerable improvements in the conversion rates of businesses, in some cases, disproportionately huge results compared to the changes made. It seemed like magic.

The blind lure of A/B testing is the possibility of a disproportionate improvement in conversions, even with minor changes, like this one.

With time, the industry has matured a lot and A/B testing is understood for what it really is. The testing part of a much more process-driven optimization plan that delivers consistent results.

Here’s a look at the conversion maturity model from the Conversion Rate Optimization report 2015 released by eConsultancy.

Conversion Rate Optimisation Maturity Model

Here are the 3 cardinal sins that are often forgotten among the more common ones:

  • Stopping testing after a failed test

  • Running multiple tests at the same time with overlapping traffic leading to skewed results

  • Not running a test for a full week, leading to bad sampling and unreliable results

Learn Phase - How to Analyze A/B Test Results

It is the analysis phase of an optimization plan that helps you close the loop for conversion optimization and fuel further optimization efforts.

Unfortunately, often optimizers look at a test result only to see if there was a winning variation. If there is one, they deploy it or if the test fails, they go back to creating more hypothesis. However, optimizers need to look deeper than that.

Consider a testing scenario. There are two possible outcomes of a test.

When a Winning Variation is Found

Great! What you should do now is answer these questions:

  • What is the cost of deploying the change (engineering hours, design hours)?

  • Does the expected increase in the revenue justify the cost involved?

Yes

  • Talk to your engineering and design teams to get the change implemented.
  • Analyze the test data to see if there are further opportunities to optimize.
  • Use these learning outcomes to fuel further optimization efforts.

No

  • Hold on to deployment.
  • Use post-test segmentation: can the hypothesis be refined for more impact?
  • Reconstruct the hypothesis.
  • Use these learning to fuel further optimization efforts.

When a Losing Variation is Found

Make sure you:

  • Look at the research; ensure that the hypothesis isn’t faulty.

  • Analyze the test data; do segmentation to reveal further insights.

  • Validate research data with surveys and visual analytics.

  • Go through relevant case studies; it could reveal new perspectives.

  • Reconstruct the hypothesis to accommodate new insights that were missed in the initial research.

  • Go back to testing.

Download this Guide

Get the complete guide to CRO in your inbox.

Want to run your own Conversion Optimization Program?

Get in touch with us for a personalized tour of VWO

Request Demo

Contact Us / Login

Product
Resources Home