BLOG

on Conversion Rate Optimization

Using A/B split testing to reduce bounce rate by 20% for an eCommerce store

One of the Visual Website Optimizer’s earliest beta testers, MedaliaArt is an online art gallery specializing in Caribbean and Latin America art. For the holiday season, they put up a sale where they give 5-55% discounts on all paintings. They wanted to determine the best location on homepage to put up that message so as to optimize for bounce rate. Their sales process is long (involving phone calls, multiple visits, etc.) so they chose to measure and optimize bounce rate instead of sales conversions. As a hypothesis, providing discounts must pull in more visitors to go through multiple pages on the website exploring different paintings.

However the challenge with putting up a ‘Holiday Sale’ message is where to show it. Displaying it prominently on the homepage will make more visitors notice it but some may find it too intrusive and leave the site immediately. On the other hand, putting it at a not-so-noticeable location may have no effect at all. So, what is the best position on page to display the ‘Holiday Sale’ (or for that matter any other promotional) message?

Only a split test can answer that. (In a split test, different visitors see (randomly selected) different versions of homepage). MedaliaArt setup a split test to optimize their website for bounce rate. First, they created a couple of versions of the homepage with ‘Holiday Sale’ displayed at different locations. Of all versions, following represented two extremes:

In-your-face ‘Holiday Sale’ message displayed in big, red font prominently on the homepage.

Sidebar ‘Holiday Sale’ message in small font.

Usually, split testing tools do not track bounce rate; they rather track conversion rate (percentage of visitors doing desired action). To track bounce rate instead, MedaliaArt did a neat trick. They defined a click on any link on the homepage as conversion. Thus the conversion rate of, for example, 40% corresponded to 100-40 = 60% bounce rate.

So, which variation had a better bounce rate? Any guesses?

They started the test and after two weeks got their first batch of conclusive results.

Message location Visitors Clicks (conversions) Conversion Rate Bounce Rate Reduction
Sidebar 145 35 24% 76% N.A.
In-your-face 123 49 40% 60% -21%

Clearly, the in-your-face, prominent promotional message has dramatically less bounce rate (60%) than the sidebar one (76%). The reduction in bounce rate of 21% is statistically significant (at 95% confidence level) so the In-your-face variation really represents a better version. The improvement in bounce rate means more interest by visitors in the paintings they are selling and potentially more sales. What they feared that a prominently displayed promotional message can backlash by irritating visitors didn’t really happen. Without split testing they could have never really known the optimal position of their promotional message. Now they know.

For the next test they do, there are a couple of suggestions for MedaliaArt (or any other eCommerce optimizing for promotional messages):

  • Have a variation with no ‘Holiday Sale’ messaging – if they had a variation with no ‘Holiday Sale’ messaging, it would have provided a benchmark to see the effect of the sales message, irrespective of the position.
  • Test message text also – instead of testing message location, it will be wise to see effect of text in the message as well. Maybe a message with the word discount (such as ‘55% discount on paintings this holiday season’) will work better than the default one (‘Holiday Sale’).
  • Optimize for sales or purchases – while optimizing for bounce rate is fine, a better metric would be to measure and optimize for sales, which is what really matters to an eCommerce site

What eCommerce stores learn from this case study?

Split testing is the only way the really know what will work and what won’t. Testing is essential to check assumptions related to promotional messages, checkout process, product category ordering, buy now button, etc. Be a little adventurous and test radically different homepage designs and ideas. You can always choose to include only a small percentage of traffic and can disable non-performing variations at a click of a button. So, what’s your excuse for not using split testing for increasing sales for your eCommerce store?

This case study is also appeared in form of interview at Practical eCommerce magazine as Split Testing Can Increase Conversion Rates.

 

Founder and CEO of Wingify.

Comments (9)

Leave a Comment
  1. At first glance, the “Holiday Sale” design looks like some sort of paid ad – so it’s not surprising that users would bounce off a page that appears filled with annoying ads.

    As many other tests have shown, offers and sales reduce bounce and increase conversion… so it’s scary to draw the wrong conclusions about offers + bounce from this test. Rather than (or in addition to) testing the messaging, a visual design/treatment test may be in order. Make it look less like an ad and more like a valuable offer.

    Question: Did MedaliaArt do any sort of design validation exercise prior to launching this test? (e.g., fivesecondtest.com)

  2. This study clearly showed the importance of A/B testing. The only problem I could see with the result was its statistical validity as 100 something visitors per treatment is mostly not enough.
    This could backfire, because you believe the treatment outperformed the original, so you start using it. But without statistically valid data (because of not enough of visitors per treatment) you could actually hurt your conversions.
    Anyway, good reading.

  3. @j-dawg: Yes, I agree that would be unwise to draw conclusions for this test (or in fact, any other A/B testing case study) and apply it directly to some other website. Every website is unique and every website needs to do a split test on its own to verify if the results are applicable.

    No, MedaliaArt did not do any sort of design validation exercise prior to launching this test. What is your experience with such activities? It sure sounds like an interesting activity.

  4. @Jan: the test is statistically valid. You need a lot of traffic when conversion rates are low (1% or 2%) but when conversion rates are higher (in this case 60%), a small sample size suffices the need.

    Of course, I agree, higher traffic will give you more confidence.

  5. @admin Please advise how we can employ the same “neat trick” of defining “a click on any link on the homepage as conversion”.

    I am running a similar test with you on VWO and was searching how to do this when I found this article.

    Ususally a homepage can have a lot of links so how to setup this goal is also really useful. Please advise.

Leave Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes : <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Contact Us / Login

Product
Resources Home