Using A/B split testing to reduce bounce rate by 20% for an eCommerce store
One of the Visual Website Optimizer’s earliest beta testers, MedaliaArt is an online art gallery specializing in Caribbean and Latin America art. For the holiday season, they put up a sale where they give 5-55% discounts on all paintings. They wanted to determine the best location on homepage to put up that message so as to optimize for bounce rate. Their sales process is long (involving phone calls, multiple visits, etc.) so they chose to measure and optimize bounce rate instead of sales conversions. As a hypothesis, providing discounts must pull in more visitors to go through multiple pages on the website exploring different paintings.
However the challenge with putting up a ‘Holiday Sale’ message is where to show it. Displaying it prominently on the homepage will make more visitors notice it but some may find it too intrusive and leave the site immediately. On the other hand, putting it at a not-so-noticeable location may have no effect at all. So, what is the best position on page to display the ‘Holiday Sale’ (or for that matter any other promotional) message?
Only a split test can answer that. (In a split test, different visitors see (randomly selected) different versions of homepage). MedaliaArt setup a split test to optimize their website for bounce rate. First, they created a couple of versions of the homepage with ‘Holiday Sale’ displayed at different locations. Of all versions, following represented two extremes:
In-your-face ‘Holiday Sale’ message displayed in big, red font prominently on the homepage.
Sidebar ‘Holiday Sale’ message in small font.
Usually, split testing tools do not track bounce rate; they rather track conversion rate (percentage of visitors doing desired action). To track bounce rate instead, MedaliaArt did a neat trick. They defined a click on any link on the homepage as conversion. Thus the conversion rate of, for example, 40% corresponded to 100-40 = 60% bounce rate.
So, which variation had a better bounce rate? Any guesses?
They started the test and after two weeks got their first batch of conclusive results.
|Message location||Visitors||Clicks (conversions)||Conversion Rate||Bounce Rate||Reduction|
Clearly, the in-your-face, prominent promotional message has dramatically less bounce rate (60%) than the sidebar one (76%). The reduction in bounce rate of 21% is statistically significant (at 95% confidence level) so the In-your-face variation really represents a better version. The improvement in bounce rate means more interest by visitors in the paintings they are selling and potentially more sales. What they feared that a prominently displayed promotional message can backlash by irritating visitors didn’t really happen. Without split testing they could have never really known the optimal position of their promotional message. Now they know.
For the next test they do, there are a couple of suggestions for MedaliaArt (or any other eCommerce optimizing for promotional messages):
- Have a variation with no ‘Holiday Sale’ messaging – if they had a variation with no ‘Holiday Sale’ messaging, it would have provided a benchmark to see the effect of the sales message, irrespective of the position.
- Test message text also – instead of testing message location, it will be wise to see effect of text in the message as well. Maybe a message with the word discount (such as ‘55% discount on paintings this holiday season’) will work better than the default one (‘Holiday Sale’).
- Optimize for sales or purchases – while optimizing for bounce rate is fine, a better metric would be to measure and optimize for sales, which is what really matters to an eCommerce site
What eCommerce stores learn from this case study?
Split testing is the only way the really know what will work and what won’t. Testing is essential to check assumptions related to promotional messages, checkout process, product category ordering, buy now button, etc. Be a little adventurous and test radically different homepage designs and ideas. You can always choose to include only a small percentage of traffic and can disable non-performing variations at a click of a button. So, what’s your excuse for not using split testing for increasing sales for your eCommerce store?
This case study is also appeared in form of interview at Practical eCommerce magazine as Split Testing Can Increase Conversion Rates.