VWO Testing
A/B Testing

A/B Testing Solutions to Maximize Your Website Conversions

Create and A/B test different versions of your website and its elements to continuously discover the best-performing versions that improve your conversions.

Know More
VWO GDPR Ready Badge
VWO CCPA Ready Badge
VWO G2Crowd Leader Spring Badge
VWO BS 10012 Certification Badge
SOC 2 logo
Follow us and stay on top of everything CRO

How To Leverage Bad Test Results

8 Min Read

VWO partnered with Christopher Nolan for this exclusive Masters of Conversion webinar to share his experience and emphasize the learnings of failed experimentation.

Christopher Nolan is a data-driven growth strategist at ShipBob, which is a tech-enabled fulfillment service company. Following are the key takeaways from the session:

Download Free: A/B Testing Guide

Experimentation and growth strategy

Chris’ focus has been growth during the course of his entire career and he deems experimentation and insights fundamental to grow in any industry. He advocates building an essential experimentation ecosystem as a foundation to test, track, and analyze your test results with confidence backed by data. The results could either be good or bad.

Key learnings from bad test results

  • When you build a strong foundation for your testing by identifying the correct combination of tools, you will have confidence in your data. 
  • It’s not enough to call a test a failure when it doesn’t meet your KPIs. You need to take a step back and analyze why something that everyone agreed had benefits failed. When you do this, you will derive a ton of value that will guide your subsequent tests. An integrated testing system with analytics and tag management tools will help you with this analysis and bring forward important insights.
  • As with failed tests, it is necessary to analyze the ‘why’ of test results even when a test is a success. Don’t take the results at face value and dig deeper into what got you there.
  • When you’re communicating test results to the leaders in your company, show them the confidence you have in your data. Talk about your test results against the KPIs, and the insights you derived from the same. Be prepared with the next steps; tell them what you plan to do with these insights. With time, teams will be as happy with insights, as they are with positive results.
Blog Banner How To Leverage Bad Test Results

Pros and cons of A/B testing set-ups

An efficient experimentation set-up comprises an A/B testing tool to test your hypotheses, integrated analytics to gauge user behavior and site engagement, and a tag management tool which is a developer-less code implementation software that allows you to inject scripts.

Pros

Cons

  • You can only track user-defined goals, which means you need to define a goal before the test in order for it to be tracked. 
  • There is some restriction on what you can track in your goals and the number of ways in which you can slice your reporting is also limited.

Advantages of CMS and Web Analytics integrated testing tool

  • It is easier to track customer data on the web pages you are testing by installing the tag manager high on the page.
  • It is easier to install tag managers within a CMS like WordPress, without needing a developer per se, as CMS offer plenty of plugin options.
  • You can form segments and custom dimensions within the CMS if it is integrated with Google Analytics, which can help track user behavior to achieve your business goals.

Biggest failures led to the biggest testing wins

The BIG failed test by Chris was on BigCommerce’s pricing page. And that led to Chris’ biggest testing win.

Chris took what he calls as the “big swing” by redesigning the pricing page at BigCommerce, a SaaS company. The “big swing” failed as it resulted in a 4% decrease in the primary KPI – the trial conversion rate, and a 25% decrease in the demo conversion rate, which was a secondary goal. However, with the help of heatmaps, scrollmaps, and navigation summaries, Chris gathered insights that helped him find what was impacting the KPIs. These insights were:

  • Mobile users liked the new design 

Using scrollmap, it was found that visitors were finding it easier to scroll the pricing page to see all the features and plans before they made a decision in a shortened version for the mobile variation.

Mobile View of The BigCommerce Pricing Page
The ‘user-friendly’ mobile variation
Image Source: BigCommerce
Desktop View of The BigCommerce Pricing Page
Desktop variation 1
Image Source: BigCommerce
  • The payment link is incredibly important

It was found during the navigation summary of the control version that visitors were clicking on a small link (the ‘Learn more’ link shown in the image below) by scrolling down. This link was removed in the variation.

Screenshot Of The Control Version Of The Mobile View

The link ‘Learn more’ became the “money maker” (revenue maker) for the pricing page as it drove massive taps
Image Source: BigCommerce
  • Toggles between Monthly and Annual plan modes drove more taps

The monthly-annual toggle on the pricing page drove more taps and scroll depth that correlated with trial conversion that did occur. However, this correlation could not be deduced from the heatmap as it only showed user behavior inferring that the KPIs were not met.

Monthly And Annual Plan Modes On Mobile View
 The revenue maker toggle was driving more taps, but it was not impacting the trial conversion rate
Image Source: BigCommerce

These insights led the team to create a V2 of the pricing page. This page had the toggle placed at the left bottom of the page and payment link retained from the control, based on the navigation summaries of the pricing page.

It saw a 15% lift in the trial conversion rate and 45% lift in the demo conversion rate.

Impressive, right?

Screenshot Of The Variation 2 On Mobile
Desktop variation 2 that took the KPIs by storm!
Image Source: BigCommerce

The successful test of pricing page at ShipBoB

Previous learnings anchor your next experiment – when Chris joined ShipBob, the first thing he looked at based on his experience was once again the pricing page.

The form abandonment rate at ShipBob’ pricing page was 30%. Instead of testing the ‘Get started here’ call-to-action(CTA) button, which was placed above the fold and was the main CTA, he ran a test for the CTA copy ‘Request a quote’ link

This link was placed in a no man’s land at the bottom right of the page, and 11% of all users landing on that page were tapping this tiny link.

Chris’ learnings at BigCommerce paid him well. He chose to make ‘Request a quote’ link as the main CTA instead of ‘Get started here’ for creating accounts. It resulted in a massive 104% lift in their new ‘Request a quote’ link clicks, using link tracking in Google Analytics. 

Screenshot Of Events Tracking in GA
That’s how you set up link tracking
Image Source: Google Analytics

His variation was a winner. However, learning from the insights, which included intensive study of the navigation summaries, tracking link, and scrollmaps data of the page in the analytics, led him to take a decision to maintain the conversion rate. 

After a successful test, he found a link that he thought of getting rid of from his winner variation (shown in the image below). But through link tracking events configured in GA, it was found that users were still interacting with that small link at the bottom of the page, contributing to the overall conversion.

Insights From The Control Version Of Shipbobs Pricing Page
Winner variation of ShipBob’s pricing page showing a tiny link that was still driving taps

Download Free: A/B Testing Guide

Effectively communicate test results to the leadership

  • Highlight the importance of the test and your main KPIs.
  • Bring on the table the key insights from the test results.
  • Show your variations in the form of images (screenshots) with KPIs, followed by an insight, if you have one. For example, in Chris’ case at BigCommerce:
Main KPIs and Results For The Test Run On Bigcommerce Pricing Page
KPIs and results with screenshots
Insights From The Experiment Run On Bigcommerce Pricing Page
One of the crucial insights from the pricing page experiment at BigCommerce
Image Source: BigCommerce
  • Show the methodology if you have Product Heads and Chief Technical Officers in the room. For example, if they ask you how you tracked the link, you can show them the event you created along with the data in analytics.
  • Link your data to your insights for credibility: If you have the Google analytics data pulled into an Excel spreadsheet, link your insight to it. It may be messy or raw but that is going to give you confidence that you’re not pulling these numbers out of thin air. 
  • Talk about specific successive plans to answer their ‘What’s next?.”

Learn from GTM and GA in test results

Use Google Tag Manager and Google Analytics in your testing to their full potential by digging into and deriving insights from:

  • Navigation summaries
  • Video engagement
  • Chatbot engagement
  • Scrollmap and link tracking
  • Form fill engagement
  • Custom dimension by pushing GA client ID

VWO and testing 

VWO produces tools that help businesses adopt experimentation and fosters a company-wide culture of experimentation within. Everyone has to go through the beginner phase while embracing the testing culture and encounter more failures than wins. But that’s where the true value and potential for growth lie. Don’t let the failed tests discourage you, instead observe them keenly to look for opportunities posing unbound potential that’s waiting to get tapped. 

End Banner How To Leverage Bad Test Results
Nida Zehra
Nida Zehra As a marketing professional and storytelling enthusiast, I prioritize understanding data and insights over everything else while crafting a new story for experimentation. Besides work, you can find me reading fiction and, at times, metamorphosing into an artist through some discerning sorcery, which leads me to maintain an art and poetry blog on Instagram. Towards an innate linguistic inclination, I am currently looking forward to learning Persian to read and understand layers of the 14th-century Iranian poet Hafez's work—one of my most treasured literary possessions.
Share
Related content
More from VWO on A/B Testing
A/B Test Duration Calculator [Free Downloadable Excel]

A/B Test Duration Calculator [Free Downloadable Excel]

In a previous post, I provided a downloadable A/B testing significance calculator (in excel). In…

Read More
Paras Chopra

Paras Chopra

3 Min Read
What is Multivariate Testing? When And How You Should Use It

What is Multivariate Testing? When And How You Should Use It

Multivariate testing (MVT) is a form of experimentation wherein multiple elements of a webpage are…

Read More
Shubhi Ahluwalia

Shubhi Ahluwalia

8 Min Read
5 A/B Tests Jagex Aced to Skyrocket Conversions for Runescape

5 A/B Tests Jagex Aced to Skyrocket Conversions for Runescape

First established in 2001, Jagex has cultivated a long-standing reputation for crafting some of the…

Read More
Sharan Suresh

Sharan Suresh

13 Min Read

Deliver great experiences. Grow faster, starting today.

Start Free Trial Request a Demo
Shanaz Khan from VWO

Hi, I am Pratyusha from the VWO Research Desk.

Join our community of 10,000+ Marketing, Product & UX Folks today & never miss the latest from the world of experience optimization.

A value for this field is required.

Thank you!

Check your inbox for the confirmation mail