BLOG

on Conversion Rate Optimization

10 Kickass A/B Testing Case Studies From Our Archive

With more than 150 case studies on successful A/B tests performed (woah!), the VWO resource library offers a plethora of actionable tips on conversion rate optimization — to newbie and veteran marketers alike.

However, going through all of those case studies at once can be a strenuous task.

So, I decided to curate the top 10, the learning from which would help you ace your next A/B test.

The following list will not only provide you with interesting test hypotheses, but also with insights on the psychology of visitors and why one website variation outperformed the other(s).

(You can also check out this post’s repurposed slideshare deck, and save more time.)

Here goes.

1. Removing Promo-code Boxes from Checkout Page Increased Total Revenue by 24.7%

Test carried out by: Bionic Gloves, an online store that designs and sells a range of  sports, military, and casual gloves.

Element(s) tested: ‘Gift Card’ and ‘Special Offer’ code boxes on the checkout page.

Hypothesis: Doing away with promo-code boxes on the checkout page will result in higher sales and lower cart abandonment rate.

Test: The test was run on 1400 visitors for a period of 48 days.

A/B test for Bionic websiteComparison image.

Result: The total revenue increased by a handsome 24.7% and Revenue Per Visitor (RPV) by 17.1%.

Analysis:
The promo-code boxes on the original page were leaking customers out of the order process!

On finding the ‘Gift Card’ and ‘Special Offer’ code boxes just before the payment page, visitors were prompted to leave the checkout process and search for the discount-laden promo-codes elsewhere.

David from Sq1 (the agency that carried out the test) rightly pointed out:

Anytime you leave the door open for a user to leave the conversion funnel — even if it seems like they’d come right back — you risk losing sales.”

This thought was further supported by Google’s auto-complete searches for ‘Bionic Gloves’:

A/B test for Bionic website - Google search 1 A/B test for Bionic website - Google search So, people were in fact looking for the promo-codes!

The variation offered no such diversion to the visitors and outperformed the control.

Tip: Providing the available discount codes to visitors in advance (on the product pages) can be an alternate solution to the problem. This is how Myntra, an Indian fashion eCommerce store, does it in style:

Promo-code by Myntra

Key Takeaways:

  • Keep your checkout page free from distractions.
    You could even remove navigation links, if seems fit. (Test it, maybe?)
  • Offer applicable promo-codes on product description page itself.

Find the full case study here

2. Introducing Hottest-selling Services on Homepage Boosted Engagements by 40.87%

Test carried out by: MedienReich ComputerTrainings, a German company that offers various software training courses.

Element(s) tested: Service offering on the homepage.

Hypothesis: Replacing ‘course categories’ with ‘best selling courses’ on the homepage will boost engagement.

Test: The test ran for a duration of 20 days.

A/B test for Medien Reich website

Comparison image.

Result: The engagement on the homepage increased by a robust 40.87% at a confidence level of 99.9%. The year-on-year value of the homepage also increased by 106.42%.

Analysis:
The variation won because it served visitors with the right information at the right place.

What are the chances someone’s course of choice is present among best-selling courses? People who know what course they want to pursue would directly search for it, if a search box is present. But there are also people who aren’t sure what they want to take up and for them ‘best-selling’ courses would be a good place to look for courses that are already popular and are of interest to them.

After looking at the courses on offer, the next big concern for the visitors will be ‘prices’. The CTA on different course boxes on the page — saying “Kurse anzeigen”, or “Show rates” in English — takes care of it.

As looking up courses and their prices became more convenient, visitors engaged more with Medien Reich.

Key Takeaways:

  • Lessen the number of clicks required for a conversion.
  • If possible, give a place to your star-performing products/services on homepage.

Find the full case study here

3. Displaying Competitors’ Higher Prices for Specific Products Made Conversions Jump by 10%

Test carried out by: Paperstone, a UK based eCommerce website that deals in office supplies.

Element(s) tested: Price information on product page.

Hypothesis: Displaying competitors’ higher prices on product pages will increase clicks on ‘Add To Basket’ and overall website conversion rate.

Test: The test was run on more than 12000 visitors for a period of two weeks.

A/B test for Paperstone website - Control

Control

A/B test for Paperstone website - Variation 1
Variation 1

A/B test for Paperstone website - Variation 2
Variation 2

Result: Conversions increased by a significant 10.67% at a confidence level of 95%.

Analysis:
Paperstone made two variations of the control and both were carrying a competitors-prices module.

(Note that this module was provided on just 5000 product pages from a total of over 18000. Paperstone had the price advantage only on these 5000 products.)

Variation 1 couldn’t perform better than the control.
This happened probably because the competitors-prices module didn’t have a differentiating presence on the page. The module was a plain text version and might have become a victim of ‘banner blindness’.

The second variation worked much better and won over the control.
The competitors-prices module was jazzed up — Paperstone’s price was highlighted against the competitors’ prices, and a header text was added. The module was also repositioned below the ‘Add to Cart’ button in a bid to clear up the CTA’s surrounding space.

With a more resounding presence now, the module efficiently displayed Paperstone’s advantage and triggered conversions.

Tip: If you are planning to introduce a similar feature on your website, here are a few points (from Paperstone) to consider:
— Would displaying competitors’ prices drive potential customers to your competitors, if the saving is only small? Or if visitors were not previously aware that the competitors sold the product?
— Would you appear more expensive on product pages that do not have the competitor price module?
— Would adding the competitor price module to your product detail pages create poor UX which distracts and puts off your users?

Key Takeaway:

  • Highlight your competitive advantages to effect a conversion.

Find the full case study here

4. Placing Trust Badges Above-the-fold and Tweaking CTA Button Increased Conversions by 32%

Test carried out by: White Card Courses, a firm that provides induction training to workers operating in the construction space across Australia.

Element(s) tested: Trust Badges and CTA on homepage.

Hypothesis: Placing trust badges above-the-fold on the homepage and changing the color and text of the CTA will result in greater conversions.

Test: The test was run on 6585 visitors for a period of three weeks.

A/B test for White Card Courses website
Comparison Image.

Result
: There was a robust 32% increase in visits to the payment page and a 20.9% increase in clicks on the payment page — at a confidence level of 99%.

Analysis:
This is why visitors liked the variation more:

  • The variation gave better visibility to the company’s differentiating factors — money back guarantee, Australia-wide recognition, and same day certificate dispatch — are prominently displayed above-the-fold on homepage. They instantly helped visitors see White Card Courses as a trustworthy brand.
  • The variation boasted a more relevant and notable CTA. With changed background and text colors, the CTA button and its subheading lie in contrast with other website elements — standing out in the visitors’ eyes. Additionally, the new benefit-oriented CTA text (“Start Now”) connects better with the visitors who are looking to ‘start’ a construction training course.

Key Takeaways:

  • Position your differentiating factors on homepage prominently — preferably above-the-fold.
  • Provide a visual contrast between your CTA and its surrounding elements.

Find the full case study here

5. Removing Drop-down Menu for Products Soared Revenue by 56.43%

Test carried out by: Body Ecology, an eCommerce store offering health-related products.

Element(s) tested: Drop-down menu for products on the homepage.

Hypothesis: Presenting a product category page in place of a drop-down menu will improve sales.

Test: The test ran for a period of two weeks.

A/B test for Body Ecology website
Comparison image.

Result: The variation improved the revenue figure by 56.43%, translating into a sizable increase of $8880.

Analysis:
It is a common practice among eCommerce stores to offer drop-down menus to display products and product categories. They do it almost religiously. Consequently, when a site performs better without a drop-down menu (as in this case), people are set to be taken aback.

The case of Body Ecology, here, is a little different from the usual.

Body Ecology deals in a range of products that includes fermented foods, probiotic culture starters, body ecology core programs, and more. As this product range is rather uncommon, visitors find it difficult to decide which product(s) are best suited to their needs.

On the control page, the drop-down menu listing only the products’ names didn’t help the visitors much. It left many visitors with only guess-clicks to find a suitable product.

In contrast, the variation page displays product images along with their descriptions. This key information makes it easy for visitors to find themselves a good fit. Following this, they happily proceed to make a purchase.

Key Takeaways:

  • Make your product information easily available to visitors.

Find the full case study here

6. Minor Change in Header Image Realized 7.8% more CTA Clicks

Test carried out by: VVAA, a professional association of over 75,000 Dutch healthcare professionals.

Element(s) tested: Header image on conversion page.

Hypothesis: Removing a horizontal line (lifeline) from the header image will increase the clicks on CTA.

Test: The test was run on 7885 visitors for a period of 20 days.

A/B test for VVAA website
Comparison image.

Result: The number of CTA clicks rose up by 7.8% at a confidence level of 99.93%.

Analysis:
The control with the lifeline graphic lost against the variation.

As it turned out, the lifeline distracted visitors’ attention away from the real CTA.

The horizontal lifeline contained multiple ‘bubbles’, one of which was sitting right underneath the CTA.

A/B test for VVAA website - bubble

The bubble had the same orange color as the CTA button, and it sported a small text below that translated to “Choose your care”. There was also an arrow-like element pointing directly towards the center of the bubble.

These attributes added to the bubble’s false importance, and possibly drove some visitors’ attention towards it (and away from the CTA).

The lady in the image, too, contributed to the bubble’s false significance.
A/B test for VVAA website - Lady's gaze

It is well known that we, humans, follow other humans’ gaze during communication.
And the lady’s gaze in the image is pointing directly towards — no points for guessing — the bubble!

With all the distractions removed, the variation page won against the control.

Key Takeaway
:

  • Declutter the space around your CTA.

Find the full case study here

7. Revamped Landing Page Increased Monthly Revenue by $106000

Test carried out by: BrookdaleLiving.com, a company that offers various community living solutions for senior citizens.

Element(s) tested: Landing page variations.

Hypothesis: A more ‘rich’ landing page will result in increased conversions.

Test: The test was run on over 30000 visitors for a duration of two months.

A/B test for Brookdale websiteComparison image.

Result: The variation with image won and increased conversions by 3.92% at a confidence level of 99.99%. The monthly revenue also rose by $106000.

Analysis:
Since the original landing page (control) was an all-text version, the scope for design improvement was immense.

Both the variations, accordingly, incorporated text and visual content smartly.
They carried the same layout and content, except that one of them displayed an image of an elderly woman and the other showed a small video of elderly customers talking about their positive experience with Brookdale.

The variation with image worked exceedingly well. The page offered much greater value to the visitors through its newly added features — testimonials, USPs and credibility logos — and pushed them for a conversion.

However, the variation with video (a favorite of many involved in the test) couldn’t outperform the control.

Matt from Brookdale explained, “our customers perhaps aren’t interested in marketing videos so much as they want to convert quickly and easily on the page.”

Using Alexa and Pewinternet, it was also found that the majority of Brookdale’s target audience had a slow-speed internet connection. It is possible that because of the slow internet, the visitors couldn’t watch the video.

Key Takeaways:

  • Display your credibility features to visitors clearly.
  • Use images or videos according to your target audience’s preference.

Find the full case study here

8. Multivariate Test on CTA Text and Form-title Resulted in a 9.1% Higher CTR

Test carried out by: Provident Hotels & Resorts, a premier organization offering room reservations in hotels, resorts and condos across Florida.

Element(s) tested: Title of conversion-form and CTA text.

Hypothesis: More ‘relatable’ text on the conversion-form and the CTA will help increase the number of form submissions.

Test: The test was run on over 27500 visitors for a month.

Multivariate test for Provident Hotels website
Original version.

Variations
All combinations.

Result: The winning variation registered a 9.1% improvement in CTA’s click-through rate.

Analysis:
Out of the above-mentioned variations, ‘Reserve a Room’ + ‘Search’ combination (#12) emerged as the winner.

The original form title, ‘Make a reservation’, didn’t clarify the action effectively. Visitors could wrongly consider it a restaurant reservation form, and bounce from the page.

The original CTA, too, had problems. It didn’t reflect the visitors’ choice of action properly.

Consider this:
The visitors reach the CTA button and read ‘Check Availability’. Some of them might think,

“Shouldn’t I get a list of hotels first, and then get to check the availability for my place of choice?”

This can leave visitors confused.

The title ‘Reserve a Room’, however, avoided ambiguities and directly talked about the action that is specific to hotels.

Similarly, the ‘Search’ CTA offered a more appropriate label for the action that follows.

Key Takeaway:

  • Make your CTA text reflect visitors’ choice of action.

Find the full case study here

9. Replacing Stock Image with a Real Picture on Homepage Led to a 161% Rise in Conversions

Test carried out by: 160 Driving Academy, an Illinois based firm that offers truck-driving classes.

Element(s) tested: The main picture on website’s truck-driving classes page.

Hypothesis: Replacing the stock image with picture of an actual student on the conversion page will lead to more enquiries.

Test:

A/B test for 160 Driving Academy website
Comparison image.

Result: Conversions on the page increased by a staggering 161%, and registrations by 38.4% — both at a confidence level of 98%.

Analysis:
The control had the following issues:

  • The control’s stock image wasn’t unique. For instance, a reverse image search on Google revealed 175 other web pages that flaunted the same stock image. Many of these pages belonged to direct competitors. The firm’s product differentiation strategy, right there, took an arrow to its knee!
  • The stock image was somewhat irrelevant. The image displayed a guy sitting casually inside a truck. He looked neither like a coach nor like a student. The picture didn’t add much to the visitor’s knowledge about the academy.

The variation offered greater value, in the following way:

  • New image in the variation looked credible. The guy in the new picture seemed like a student from the academy. The image also offered a preview of a truck that is used for training the students. All in all, the new image built up the academy’s credibility.
  • The new image offered subtle branding and a hidden CTA. In the control page, there was only one instance where “160 Driving Academy” is mentioned. With the new image, the variation contained three such instances. The recall might have helped some customers return to the site and register for the course. In addition, the color tone of the new image perfectly matched with that of the overall site. These factors could have guided the variation’s higher number of conversions.

Key Takeaways:

  • Make sure your pictures (stock images included) are unique, or of real stakeholders — employees, students, etc.
  • Use pictures that directly signify your business (or the specific webpage).

Find the full case study here

10. Adding Threshold for Free Shipping Increased AOV by 7.32%

Test carried out by: NuFace, an anti-aging skin care company with an active online store.

Element(s) tested: ‘Free shipping’ text element on homepage.

Hypothesis: Providing ‘free shipping’ incentive to visitors will improve sales.

Test:

A/B test for NuFace website

Comparison image.

Result: The total orders increased by a whopping 90% at a confidence level of 96%. The Average Order Value, too, rose by 7.32%.

Analysis:
The magic word ‘Free’ worked. Again!

If you’re still skeptical, and doubt the effectiveness of the free shipping incentive (as it can only be availed after shopping for $75 or more), please bear with me a little longer.

Take a look at this screenshot from their products page:

A/B test for NuFace website - products

The screenshot contains products that lie in a lower price range ($10-30) as well as  products that are deemed expensive (>$300).

Considering the diverse product range and price brackets that NuFace has, the threshold has been very cleverly (and aptly) set at $75.

Provided with the option of buying less expensive products, customers could easily make an impulsive buy to reach the $75 threshold and avail of free shipping.

Further, the incentive encourages customers to go for premium products too. Free shipping can push ‘modest buyers’ to try out the expensive product range — something they have probably wanted to do for a long time.

Going with any of the two choices, the visitors are set to increase the total orders.

Key Takeaway
:

  • Try offering (conditional) incentives to your customers for increasing sales.

Find the full case study here

I hope these case studies gave your brain a good tickle; even help you manufacture new test ideas. Let me know which case study amused you the most in the comments section below.

Nitin is a traveler, a cinephile, and a webaholic. (He just can't get enough of cat videos!) Professionally, Nitin is a marketer at VWO, who loves to write about Conversion Optimization. Find him on Twitter: @NitinDeshdeep

Comments (5)

Leave a Comment
  1. Thanks for this article! I’m totally going to read full versions of the case studies listed here. There’s just one thing about A/B testing that I’d like to mention. What sometimes happens is that despite the fact that tests deliver conversion rate lifts, they fail to deliver a real impact on business. That’s why not focusing exclusively on conversion rate during A/B testing may be a good idea. If you want to find out more, take a look at this article:
    http://www.mavenec.com/blog/dont-focus-on-conversion-rate-during-your-next-ab-test/?utm_medium=social&utm_source=comments&utm_campaign=20150720_CommentCampaign

    Greetings!

  2. These are awesome case studies, but what about you?

    I guess that you have been doing a lot of change to your own website and testing different landings, CTA Button, headlines, etc.

    1. Hi,

      Yes, we try to optimize our website as much as we can.
      In fact, we have been doing a lot of A/B testing (with CRO tricks) here on our blog.
      We might post a case study on it soon. 🙂

  3. This is a really interesting article and particularly so because we have a website in a similar industry to one you have used as a case study. Our website is information based, not so heavily focused on sales, but seeing an increase of 32% to the payment page is a huge result. I’d be fascinated to know what kind of improvements can be seen on an information based site for getting people to spend more time on the site/consume more content. Trust symbols are clearly important, but when it’s not payment related, it can be a little bit harder to use trusted symbols. Thanks for such an insightful article Nitin, much appreciated.

    1. Hi Jaimee,

      Glad you found the article useful.

      Regarding your concern about improving information-based websites, we’ve seen that customer testimonials and other social proof elements often tend to improve conversions. Of course, the design, readability, and content quality of the website matters greatly, too.

      Run A/B tests on all these aspects of your website, and find out which version works best. Do let us know about the results. 🙂

Leave Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes : <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Contact Us / Login

Product
Resources Home