This post is a compilation of best A/B split testing case studies available on the Internet. In a way, this compilation proves that: a) A/B split testing really works; b) it doesn’t take much effort (hello Visual Website Optimizer); c) you should be doing it right now! I think case studies are an interesting way to learn about a new domain. So, consider this post as a starting point for the wonderful world of A/B split testing. Let’s start the list of case studies which you should definitely read today.
37Signals did an A/B test on the headline of their pricing page. What they found was that “30 day Free Trial on All Accounts” had 30% more signups than the original “Start a Highrise Account”.:
Lesson: in a headline, the word “free” coupled with specific benefits (“30 day trial” v/s “start an account”) can do wonders.
The much hyped split testing case which involved testing of multiple different versions of call to text for twitter followers. Dustin found out that “You should follow me on Twitter here” worked 173% better than his control text “I’m on Twitter”. Though his results were fantastic, I recently wrote a post titled “You should NOT follow me on Twitter” arguing why you should be wary of applying this results without testing on your website.
Lesson: persuasive call to action “you should follow” proves to be much better than a passive call to action “i’m on twitter” (well, why would a visitor care).
A simpler variation which had less options for a visitor to choose from resulted in 20% increase in conversions. The winning version was also much easier on eyes (in terms of details and text) as compared to the control.
Lesson: too many options on a page usually overwhelms the visitor and s/he simply bounces off the page.
Reducing a two page checkout process to one page checkout increased conversions by 20%. Though overall increase in sales and orders didn’t increase much, checkout % did increase.
Lesson: lesser the steps you have in your funnel, higher will be your conversion rate. A lot of visitors drop off at each stage of conversion funnel so try to shorten it as much as possible (ideally to one page).
Sensationalist headline but interesting results. hey tried different combinations – ads in left v/s right; ads top v/s bottom; test ads v/s image ads. Not sure if the results are statistically significant, but definitely a good case study.
Lesson: proves that A/B testing not just works for product/sales websites, but can work for advertisement supported pages too.
They demonstrated a 258% in opt-in rate. Focus was to remove all distractions and require the visitor to only provide email address. For completing his/her complete profile, the landing page incited the visitors with an Amazon gift card (which was again split tested).
Lesson: require the visitor to input minimal amount of information on the landing page, you can always ask for more information later (it related is the concept of sunk cost).
In addition to A/B split testing, they used a lot of other tools/techniques – clickmaps/user feedback/usability testing. The final result was an increase in 400% increase in conversions.
Lesson: simplicity; social-proof; role of video in conversions; clarity in design
Skype tested their homepage for 3 different variations. Key difference between the three versions was the level of details presented on the page. Guess which variation won the test? The simplest variation of all (see image above) saw an increase of 5% in downloads. The figure may not look like much of an improvement but when you consider that Skype may be getting tens of thousands of downloads each day, this increase becomes quite large and totally worth-it spending thousands of dollars on doing the A/B test. (But, hey, you don’t have too – after all, Visual Website Optimizer is free to signup).
Lesson: simplicity, simplicity and some more simplicity
Do you know of any other interesting A/B split testing case studies which I should add in the post? Perhaps you may want to take help of A/B ideafox (a search engine for A/B split testing case studies).