Last week, we ran a contest (with $3000 worth prize) where we asked you to predict which variation in an A/B test produced 400% increase in conversions. We received over 50 entries and (ironically) the split of answers was roughly 50/50. This shows just how hard it is to predict A/B test winner in advance. Only real data tells the truth! For this case, now you have the answer: the variation that DID NOT have security badge actually increased conversions by 400%. This is in stark contrast to one of our earlier A/B testing case studies where a variation which had security badge increased conversions by 32% (over the control which did not have such badge). This shows that every site is unique and best practices shouldn’t be adopted without doing any testing to prove that they would work for your site too. Out of all the correct entries, we’ve selected one entry randomly to win this contest:
The $3000 worth prize (twelve month Visual Website Optimizer $249/mo subscription) goes to: Subhash Surampudi [link to his comment]. Congratulations to the winner!
Case Study ICoupon Blog is a coupon website that has been around for about 5 months and provides coupons and rebates for different services and products. They decided to use Visual Website Optimizer to increase click-throughs on these coupons. What they tested was whether having a large ‘Secure’ icon in the sidebar would help with our conversion rate. They thought the icon would help because it would push down other (distracting) text in the sidebar. [Note to ICoupon: a great follow up test would be to further test this hypothesis by having a variation where sidebar text is completely removed] Here are the variations they tested:
– Version A: with Security badge in sidebar –
– Version B: without Secure badge in sidebar –
Reasons why this particular variation was tested In the words of Bradley Spencer (from ICoupon):
Actually I thought the ‘Secure’ icon would win hands down. It was my idea and I wanted to prove to my partners that it was a good idea. I was really surprised to learn that it didn’t help at all.
They also had some user testing done and everyone seemed to notice the ‘Secure’ button a lot. It was one of the most prominent parts of the page. So they think removing it helped focus the user on they we really wanted them to push. A/B test results They found that without the secure icon had over 400% improvement on conversions as compared to having the image. [Note: results are statistically significant] Strangely, removing the icon dropped site engagement by about 17%. However, with their business model a ‘bounce’ is actually a good thing, so site engagement isn’t as large of a factor. In nutshell, in this particular case, it looks like the secure icon distracted the visitor from the main focus of the page. Lessons As Bradley sums up the lessons for others:
Make each page designed to get the user to do one thing, and try to focus all of their attention on that one thing.
He also adds a word about the tool they used for the test:
Visual Website Optimizer was invaluable. We would have kept wasting user’s attention with that icon without VWO.