VWO Logo
Request Pricing
Dashboard
Like this post?
Read our in-depth guide to A/B Testing
10 Min Read

How to Create a Strong A/B Testing Hypothesis?

Nida Zehra
Nida Zehra
Content Marketer at VWO

“That is one small step for a man; one giant leap for mankind.”

These were the iconic words of Neil Armstrong as he became the first human ever to set foot on the soil of a land that did not belong to earth! 

Now step back for a second and think, what all planning, trials, hypothecation, and testing would have gone to ensure Neil took that small step from his lunar landing module to the powdery surface of the moon? It was clearly not a leap of faith!

Be it this fascinating journey that took a million miles or a minute-long experience of a visitor landing on your website–every journey requires a lot of testing and hypothecation in form of small steps to find the optimal route.

You could be an enterprise, a mid- or a small-sized organization, freelance marketer, or even the next space mission! 

Running a test without a hypothesis is like getting lost in a labyrinth. It will lead you somewhere but not to your desired destination, which makes it an absolute necessity before making any decision to move from point A to point B.  

In this post, you will learn about constructing strong A/B testing hypotheses, which will make you create your own story a fascinating one.

What is a Hypothesis?

A prediction that you make before running a test is called a hypothesis. It is a bold statement that clearly states what change do you want to make, why do you want to so, and its expected impact.  

We have emphasized enough on why constructing a hypothesis is vital before running any test. Now, let us make it more clear following an example here.

One day you wake up and want to run a test for the color of the CTA button at your website. 

You rush and achieve conversion lifts with this change, but you do not know what exactly was that you wanted out of this test? 

You did not pay any heed to your conversion funnel; you did not analyze the test holistically before executing it!  

There is a difference between running a test scientifically to prove something and a test that generates random results.

Now, this something is your hypothesis.

Components of a Hypothesis

As you would notice here, in the world of user experience optimization, a strong hypothesis is made up of three components, defining a problem, describing a proposed solution, and measuring results.

flow diagram on the various components of a hypothesis

Think of it as a glue that ties the problem to a solution and eventually to the results. 

For instance, you could hypothesize that adding trust badges to your payment page could cater to the problem of low conversion rates on that page. You can find out why that happens by identifying the right metrics for success.

But how does one begin to start formulating a hypothesis?

How to Formulate a Winning Hypothesis?

You need not follow best practices crafted by someone else for your business. Go ahead and construct your own best practices. 

Below are some essential elements that make a solid hypothesis:

1. They aim to alter customer behavior, either positively or negatively.

A psychological principle often forms the basis of a hypothesis that triggers a reaction from prospects. 

It changes the way they perceive your offer/brand, which alters their behavior during the test. Like in the headline example below, the urgency of the new message is the reason why the variation headline is expected to perform better than the original headline. 

Changing the headline from ‘Grab your tickets now!’ to 

‘Tickets filling out soon – only last 50 left!’ could increase ticket sales online. 

Only because you follow the above syntax to formulate a hypothesis doesn’t mean that you’ve got the winning hypothesis.

2. They focus on deriving customer learning from tests.

When pros develop a hypothesis, their focus is on the big picture. Dr. Flint McGlaughlin reveals their secret sauce:

So your test might give you any result, what’s important is for you to analyze ‘why’ your visitors behaved in a certain way? 

As data-driven marketers, it might seem difficult sometimes to make your peace with negative lifts. 

But if the test reveals significant customer learning, it can pave the way for colossal conversion lifts in the future. 

For example, when Michael Aagaard, the conversion copywriter of Content Verve, conducted a CTA copy test on his client’s website. He learned that changing the CTA copy from ‘Start your 30 day free trial’ to ‘Start my 30-day free trial’ resulted in 90% increase in sign-ups. 

screenshot of the A/B test run by Michael Aagaard of Content Verve
Image Source [1]

However, he cautioned that applying this technique would not work across the linguistic globe though.

3. They are derived from at least some evidence.

You can explore many avenues to construct good hypotheses. 

The image below by the popular Conversion Expert, Craig Sullivan, shows several ways in which you can collect meaningful insights.

the several ways you can collect meaningful insights to construct a hypothesis
Image Source [2]

a. Usability Testing

In simple words, you can sit and observe how your test participants use your website. 

Make notes where they get stuck or confused. Form your hypotheses to improve these situations and let A/B tests decide if they work for you.

This method gives you exceptional insights about your customers’ workarounds and struggles in using your website. 

Write down the exact questions you would like to ask during your research. Asking questions related to homepage, checkout, pricing page, and navigation reveal great insights. 

Some sample questions you can ask are: 

  • Was it easy to find what you were looking for? 
  • Were the words/vocab used to define categories/sub-categories clear to you? 
  • Do you have any suggestions to improve our website navigation? 
  • Does our website look credible to you? 
  • Is our pricing clear? 
  • Is there anything else you’d like to know before signing up with us? 
  • Will you shop with us again? Why/why not? 
  • Do you think the form has any confusing/unnecessary input fields? 

b. Customer Surveys

Surveys are an effective tool to understand your customer segments, to know their intent/pain points or concerns which in turn help you understand the glitches on the websites, and so on. 

Here are some sample questions to use for on-site surveys:

a screenshot of the sample questions used by twilio for on-site surveys
Image Source [3]

Visiting the pricing page shows the intent of buying. See how Qualaroo leverages targeted traffic on their pricing page to understand customer pain points:

screenshot of the survey run by Qualaroo on their pricing page

Ask these survey questions during cancellation of a subscription:

screenshot of a survey asking questions to inquire about the reasons for cancelling subscription

To existing customers, you can ask: 

Have you ever criticized/praised us to someone in the past few months? 

What did you say? 

If you are made the owner of [insert your company name], what would you change? 

For those who have just signed up with you, you can send an auto-generated mail asking: 

Did you have any doubts or concerns about our product/service before you signed up? 

What made you overcome those doubts? How did you hear about us? 

Too many questions can be annoying, especially in on-site surveys. Make sure you respect your prospects’ choice if they choose not to answer your questions. 

Similarly, off-site surveys also serve the same purpose of gathering feedback but via email or third-party survey websites. 

Apart from SurveyMonkey, you could also use Fluid Surveys and Confirmit for designing off-site feedback surveys.

c. Heatmaps

The data in your testing tool can tell you what your visitors are doing but not why they are doing it. 

Heatmaps can help you identify interest areas of your prospects as well as what areas they choose to ignore. Sometimes this can help you identify great insights when an important page element is going unnoticed by visitors because some other element on the page is stealing its thunder. 

The heatmap below shows how a non-clickable element in the image takes away all the attention from the call-to-action (CTA) on the page:

screenshot of a heatmap that highlights the distraction of a non-clickable element in an image
Image Source [4]

Later, when this element was removed, the heatmap shows a clear emphasis on the call-to-action of the page (as it should be):

screenshot of the heatmap of the web-page when the distracting element has been removed

Without referring to their heatmap, TechWyse had never been able to understand the reason for their dropping conversion numbers.
You can read their complete case study
here[1]

If not this, heatmaps can sometimes also help you figure out a page element/navigation item that may be taking too much of important online real estate while being completely ignored by users. 

You can consider replacing it with something more relevant to your conversion goal and conduct an A/B test to see how it performs. 

d. Website Analytics

You might know this already, but exploring data in your website analytics can give you some great ideas to get started. 

For instance, the hypothesis of “adding trust badges on payment page” we formed earlier, could have led from a “high exit rate of the page.” The exit rate of the page — along with other metrics — can be found within your website analytics. 

Website analytics tools like Google Analytics can show you quantitative data on how visitors navigate your website on a site architecture level. 

Some of the important metrics that you could track to validate an idea and build a hypothesis are:

 

  • Traffic Report: Metrics like total traffic, the total number of visitors (overall and on individual pages) could help you track how many people will the test impact and how long it would take to finish it. 
  • Acquisition Report: This could help you determine where your visitors are coming from (your best traffic sources) and how the performance differs between different channels. 
  • Landing Page Report: Your top landing and exit pages show how visitors enter and leave the site. 
  • Funnel Report: This would give you insights into questions like where your visitors enter into or exit from your marketing funnel, and how do they navigate between the different pages. 
  • Device Type: This will help you decide whether you should focus on optimizing the user experience on a particular device on priority. 

For any observation that you come across from analyzing these, ask yourself enough number of ‘why’s’ to form a solid hypothesis.

Test with Confidence

When you have done your part of grunt work, there’s nothing to be afraid of. It is time to take that leap:

  • Every outcome should be hypothesized: Remember that every test is an opportunity to learn. Think one step ahead of your experiment as to what you would learn if your hypothesis is proven correct or incorrect.
  • High-level goals mapping: You must keep yourself acquainted with company-wide goals in order to construct robust tests. You need to ensure that the tests you are conducting are backed by data. They will provide you with the insights that would help grow your business. 
  • Documentation: This is a very crucial step. Make it a habit to document all of the tests that you run. This will not only serve as a reference for future tests but also can be utilized as a forum wherein marketers can share the context for the tests conducted so far. 

Forming a well-structured hypothesis is a critical piece in the conversion optimization puzzle. It helps you identify and remove the friction along your conversion funnel.

You might find that you have multiple hypotheses to try. But instead of running in all directions, prioritize your tests and monitor them for both conversions and learning. 


Like this post?
Read our in-depth guide to A/B Testing
More from VWO on A/B Testing

A/B Test Duration Calculator [Free Downloadable Excel]

In a previous post, I provided a downloadable A/B testing significance calculator (in excel). In…

Read More
Paras Chopra

Paras Chopra

3 Min Read

Introducing Browse Mode: create an A/B test even behind the login walls of a website

We get bored (and frustrated) if we think it has been long since we last…

Read More
Paras Chopra

Paras Chopra

2 Min Read

Rules-of-thumb for A/B and Multivariate tests

I recently got interviewed on Unbounce blog as a conversion hero. In the interview, I…

Read More
Paras Chopra

Paras Chopra

5 Min Read
Shanaz Khan from VWO

Hi, I am Shanaz from the VWO Research Desk.

Join our community of 10,000+ Marketing, Product & UX Folks today & never miss the latest from the world of experience optimization.

A value for this field is required.

Thank you!

Check your inbox for the confirmation mail