10 Kickass A/B Testing Case Studies From Our Archive
With more than 150 case studies on successful A/B tests performed (woah!), the VWO resource library offers a plethora of actionable tips on conversion rate optimization — to newbie and veteran marketers alike.
However, going through all of those case studies at once can be a strenuous task.
So, I decided to curate the top 10, the learning from which would help you ace your next A/B test.
The following list will not only provide you with interesting test hypotheses, but also with insights on the psychology of visitors and why one website variation outperformed the other(s).
(You can also check out this post’s repurposed slideshare deck, and save more time.)
1. Removing Promo-code Boxes from Checkout Page Increased Total Revenue by 24.7%
Test carried out by: Bionic Gloves, an online store that designs and sells a range of sports, military, and casual gloves.
Element(s) tested: ‘Gift Card’ and ‘Special Offer’ code boxes on the checkout page.
Hypothesis: Doing away with promo-code boxes on the checkout page will result in higher sales and lower cart abandonment rate.
Result: The total revenue increased by a handsome 24.7% and Revenue Per Visitor (RPV) by 17.1%.
The promo-code boxes on the original page were leaking customers out of the order process!
On finding the ‘Gift Card’ and ‘Special Offer’ code boxes just before the payment page, visitors were prompted to leave the checkout process and search for the discount-laden promo-codes elsewhere.
David from Sq1 (the agency that carried out the test) rightly pointed out:
“Anytime you leave the door open for a user to leave the conversion funnel — even if it seems like they’d come right back — you risk losing sales.”
This thought was further supported by Google’s auto-complete searches for ‘Bionic Gloves’:
The variation offered no such diversion to the visitors and outperformed the control.
Tip: Providing the available discount codes to visitors in advance (on the product pages) can be an alternate solution to the problem. This is how Myntra, an Indian fashion eCommerce store, does it in style:
- Keep your checkout page free from distractions.
You could even remove navigation links, if seems fit. (Test it, maybe?)
- Offer applicable promo-codes on product description page itself.
2. Introducing Hottest-selling Services on Homepage Boosted Engagements by 40.87%
Test carried out by: MedienReich ComputerTrainings, a German company that offers various software training courses.
Element(s) tested: Service offering on the homepage.
Hypothesis: Replacing ‘course categories’ with ‘best selling courses’ on the homepage will boost engagement.
Result: The engagement on the homepage increased by a robust 40.87% at a confidence level of 99.9%. The year-on-year value of the homepage also increased by 106.42%.
The variation won because it served visitors with the right information at the right place.
What are the chances someone’s course of choice is present among best-selling courses? People who know what course they want to pursue would directly search for it, if a search box is present. But there are also people who aren’t sure what they want to take up and for them ‘best-selling’ courses would be a good place to look for courses that are already popular and are of interest to them.
After looking at the courses on offer, the next big concern for the visitors will be ‘prices’. The CTA on different course boxes on the page — saying “Kurse anzeigen”, or “Show rates” in English — takes care of it.
As looking up courses and their prices became more convenient, visitors engaged more with Medien Reich.
- Lessen the number of clicks required for a conversion.
- If possible, give a place to your star-performing products/services on homepage.
3. Displaying Competitors’ Higher Prices for Specific Products Made Conversions Jump by 10%
Test carried out by: Paperstone, a UK based eCommerce website that deals in office supplies.
Element(s) tested: Price information on product page.
Hypothesis: Displaying competitors’ higher prices on product pages will increase clicks on ‘Add To Basket’ and overall website conversion rate.
Paperstone made two variations of the control and both were carrying a competitors-prices module.
(Note that this module was provided on just 5000 product pages from a total of over 18000. Paperstone had the price advantage only on these 5000 products.)
Variation 1 couldn’t perform better than the control.
This happened probably because the competitors-prices module didn’t have a differentiating presence on the page. The module was a plain text version and might have become a victim of ‘banner blindness’.
The second variation worked much better and won over the control.
The competitors-prices module was jazzed up — Paperstone’s price was highlighted against the competitors’ prices, and a header text was added. The module was also repositioned below the ‘Add to Cart’ button in a bid to clear up the CTA’s surrounding space.
With a more resounding presence now, the module efficiently displayed Paperstone’s advantage and triggered conversions.
Tip: If you are planning to introduce a similar feature on your website, here are a few points (from Paperstone) to consider:
— Would displaying competitors’ prices drive potential customers to your competitors, if the saving is only small? Or if visitors were not previously aware that the competitors sold the product?
— Would you appear more expensive on product pages that do not have the competitor price module?
— Would adding the competitor price module to your product detail pages create poor UX which distracts and puts off your users?
- Highlight your competitive advantages to effect a conversion.
4. Placing Trust Badges Above-the-fold and Tweaking CTA Button Increased Conversions by 32%
Test carried out by: White Card Courses, a firm that provides induction training to workers operating in the construction space across Australia.
Element(s) tested: Trust Badges and CTA on homepage.
Hypothesis: Placing trust badges above-the-fold on the homepage and changing the color and text of the CTA will result in greater conversions.
Test: The test was run on 6585 visitors for a period of three weeks.
This is why visitors liked the variation more:
- The variation gave better visibility to the company’s differentiating factors — money back guarantee, Australia-wide recognition, and same day certificate dispatch — are prominently displayed above-the-fold on homepage. They instantly helped visitors see White Card Courses as a trustworthy brand.
- The variation boasted a more relevant and notable CTA. With changed background and text colors, the CTA button and its subheading lie in contrast with other website elements — standing out in the visitors’ eyes. Additionally, the new benefit-oriented CTA text (“Start Now”) connects better with the visitors who are looking to ‘start’ a construction training course.
- Position your differentiating factors on homepage prominently — preferably above-the-fold.
- Provide a visual contrast between your CTA and its surrounding elements.
5. Removing Drop-down Menu for Products Soared Revenue by 56.43%
Test carried out by: Body Ecology, an eCommerce store offering health-related products.
Element(s) tested: Drop-down menu for products on the homepage.
Hypothesis: Presenting a product category page in place of a drop-down menu will improve sales.
Result: The variation improved the revenue figure by 56.43%, translating into a sizable increase of $8880.
It is a common practice among eCommerce stores to offer drop-down menus to display products and product categories. They do it almost religiously. Consequently, when a site performs better without a drop-down menu (as in this case), people are set to be taken aback.
The case of Body Ecology, here, is a little different from the usual.
Body Ecology deals in a range of products that includes fermented foods, probiotic culture starters, body ecology core programs, and more. As this product range is rather uncommon, visitors find it difficult to decide which product(s) are best suited to their needs.
On the control page, the drop-down menu listing only the products’ names didn’t help the visitors much. It left many visitors with only guess-clicks to find a suitable product.
In contrast, the variation page displays product images along with their descriptions. This key information makes it easy for visitors to find themselves a good fit. Following this, they happily proceed to make a purchase.
- Make your product information easily available to visitors.
6. Minor Change in Header Image Realized 7.8% more CTA Clicks
Test carried out by: VVAA, a professional association of over 75,000 Dutch healthcare professionals.
Element(s) tested: Header image on conversion page.
Hypothesis: Removing a horizontal line (lifeline) from the header image will increase the clicks on CTA.
Result: The number of CTA clicks rose up by 7.8% at a confidence level of 99.93%.
The control with the lifeline graphic lost against the variation.
As it turned out, the lifeline distracted visitors’ attention away from the real CTA.
The horizontal lifeline contained multiple ‘bubbles’, one of which was sitting right underneath the CTA.
The bubble had the same orange color as the CTA button, and it sported a small text below that translated to “Choose your care”. There was also an arrow-like element pointing directly towards the center of the bubble.
These attributes added to the bubble’s false importance, and possibly drove some visitors’ attention towards it (and away from the CTA).
It is well known that we, humans, follow other humans’ gaze during communication.
And the lady’s gaze in the image is pointing directly towards — no points for guessing — the bubble!
With all the distractions removed, the variation page won against the control.
- Declutter the space around your CTA.
7. Revamped Landing Page Increased Monthly Revenue by $106000
Test carried out by: BrookdaleLiving.com, a company that offers various community living solutions for senior citizens.
Element(s) tested: Landing page variations.
Hypothesis: A more ‘rich’ landing page will result in increased conversions.
Test: The test was run on over 30000 visitors for a duration of two months.
Since the original landing page (control) was an all-text version, the scope for design improvement was immense.
Both the variations, accordingly, incorporated text and visual content smartly.
They carried the same layout and content, except that one of them displayed an image of an elderly woman and the other showed a small video of elderly customers talking about their positive experience with Brookdale.
The variation with image worked exceedingly well. The page offered much greater value to the visitors through its newly added features — testimonials, USPs and credibility logos — and pushed them for a conversion.
However, the variation with video (a favorite of many involved in the test) couldn’t outperform the control.
Matt from Brookdale explained, “our customers perhaps aren’t interested in marketing videos so much as they want to convert quickly and easily on the page.”
Using Alexa and Pewinternet, it was also found that the majority of Brookdale’s target audience had a slow-speed internet connection. It is possible that because of the slow internet, the visitors couldn’t watch the video.
- Display your credibility features to visitors clearly.
- Use images or videos according to your target audience’s preference.
8. Multivariate Test on CTA Text and Form-title Resulted in a 9.1% Higher CTR
Test carried out by: Provident Hotels & Resorts, a premier organization offering room reservations in hotels, resorts and condos across Florida.
Element(s) tested: Title of conversion-form and CTA text.
Hypothesis: More ‘relatable’ text on the conversion-form and the CTA will help increase the number of form submissions.
Result: The winning variation registered a 9.1% improvement in CTA’s click-through rate.
Out of the above-mentioned variations, ‘Reserve a Room’ + ‘Search’ combination (#12) emerged as the winner.
The original form title, ‘Make a reservation’, didn’t clarify the action effectively. Visitors could wrongly consider it a restaurant reservation form, and bounce from the page.
The original CTA, too, had problems. It didn’t reflect the visitors’ choice of action properly.
The visitors reach the CTA button and read ‘Check Availability’. Some of them might think,
“Shouldn’t I get a list of hotels first, and then get to check the availability for my place of choice?”
This can leave visitors confused.
The title ‘Reserve a Room’, however, avoided ambiguities and directly talked about the action that is specific to hotels.
Similarly, the ‘Search’ CTA offered a more appropriate label for the action that follows.
- Make your CTA text reflect visitors’ choice of action.
9. Replacing Stock Image with a Real Picture on Homepage Led to a 161% Rise in Conversions
Test carried out by: 160 Driving Academy, an Illinois based firm that offers truck-driving classes.
Element(s) tested: The main picture on website’s truck-driving classes page.
Hypothesis: Replacing the stock image with picture of an actual student on the conversion page will lead to more enquiries.
The control had the following issues:
- The control’s stock image wasn’t unique. For instance, a reverse image search on Google revealed 175 other web pages that flaunted the same stock image. Many of these pages belonged to direct competitors. The firm’s product differentiation strategy, right there, took an arrow to its knee!
- The stock image was somewhat irrelevant. The image displayed a guy sitting casually inside a truck. He looked neither like a coach nor like a student. The picture didn’t add much to the visitor’s knowledge about the academy.
The variation offered greater value, in the following way:
- New image in the variation looked credible. The guy in the new picture seemed like a student from the academy. The image also offered a preview of a truck that is used for training the students. All in all, the new image built up the academy’s credibility.
- The new image offered subtle branding and a hidden CTA. In the control page, there was only one instance where “160 Driving Academy” is mentioned. With the new image, the variation contained three such instances. The recall might have helped some customers return to the site and register for the course. In addition, the color tone of the new image perfectly matched with that of the overall site. These factors could have guided the variation’s higher number of conversions.
- Make sure your pictures (stock images included) are unique, or of real stakeholders — employees, students, etc.
- Use pictures that directly signify your business (or the specific webpage).
10. Adding Threshold for Free Shipping Increased AOV by 7.32%
Test carried out by: NuFace, an anti-aging skin care company with an active online store.
Element(s) tested: ‘Free shipping’ text element on homepage.
Hypothesis: Providing ‘free shipping’ incentive to visitors will improve sales.
The magic word ‘Free’ worked. Again!
If you’re still skeptical, and doubt the effectiveness of the free shipping incentive (as it can only be availed after shopping for $75 or more), please bear with me a little longer.
Take a look at this screenshot from their products page:
The screenshot contains products that lie in a lower price range ($10-30) as well as products that are deemed expensive (>$300).
Considering the diverse product range and price brackets that NuFace has, the threshold has been very cleverly (and aptly) set at $75.
Provided with the option of buying less expensive products, customers could easily make an impulsive buy to reach the $75 threshold and avail of free shipping.
Further, the incentive encourages customers to go for premium products too. Free shipping can push ‘modest buyers’ to try out the expensive product range — something they have probably wanted to do for a long time.
Going with any of the two choices, the visitors are set to increase the total orders.
- Try offering (conditional) incentives to your customers for increasing sales.
I hope these case studies gave your brain a good tickle; even help you manufacture new test ideas. Let me know which case study amused you the most in the comments section below.