This website works best with JavaScript enabled. Learn how to enable JavaScript.

VWO Blog

A Blog Around Everything Conversion Optimization And Beyond

Never miss an update from VWO

A value for this field is required.

8 A/B split testing case studies you cannot afford to miss

This post is a compilation of best A/B split testing case studies available on the Internet. In a way, this compilation proves that: a) A/B split testing really works; b) it doesn’t take much effort (hello Visual Website Optimizer); c) you should be doing it right now! I think case studies are an interesting way to learn about a new domain. So, consider this post as a starting point for the wonderful world of A/B split testing.  Let’s start the list of case studies which you should definitely read today.

Writing Decisions: Headline tests on the Highrise signup page

37Signals did an A/B test on the headline of their pricing page. What they found was that “30 day Free Trial on All Accounts” had 30% more signups than the original “Start a Highrise Account”.:

Lesson: in a headline, the word “free” coupled with specific benefits (“30 day trial” v/s “start an account”) can do wonders.

Dustin Curtis’ “You should follow me on Twitter here”

The much hyped split testing case which involved testing of multiple different versions of call to text for twitter followers. Dustin found out that “You should follow me on Twitter here” worked 173% better than his control text “I’m on Twitter”. Though his results were fantastic, I recently wrote a post titled “You should NOT follow me on Twitter” arguing why you should be wary of applying this results without testing on your website.

Lesson: persuasive call to action “you should follow” proves to be much better than a passive call to action “i’m on twitter” (well, why would a visitor care).

Daily Burn, 20%+ Improvement

A simpler variation which had less options for a visitor to choose from resulted in 20% increase in conversions. The winning version was also much easier on eyes (in terms of details and text) as compared to the control.

Lesson: too many options on a page usually overwhelms the visitor and s/he simply bounces off the page.

A/B Test Case Study: Single Page vs. Multi-Step Checkout

Reducing a two page checkout process to one page checkout increased conversions by 20%. Though overall increase in sales and orders didn’t increase much, checkout % did increase.

Lesson: lesser the steps you have in your funnel, higher will be your conversion rate. A lot of visitors drop off at each stage of conversion funnel so try to shorten it as much as possible (ideally to one page).

How To Increase Your Adsense Earnings 94% Overnight

Sensationalist headline but interesting results. hey tried different combinations – ads in left v/s right; ads top v/s bottom; test ads v/s image ads. Not sure if the results are statistically significant, but definitely a good case study.

Lesson: proves that A/B testing not just works for product/sales websites, but can work for advertisement supported pages too.

Marketing Experiments Response Capture case study – triple digit increase in conversions

They demonstrated a 258% in opt-in rate. Focus was to remove all distractions and require the visitor to only provide email address. For completing his/her complete profile, the landing page incited the visitors with an Amazon gift card (which was again split tested).

Lesson: require the visitor to input minimal amount of information on the landing page, you can always ask for more information later (it related is the concept of sunk cost).

How we increased the conversion rate of by over 400%

In addition to A/B split testing, they used a lot of other tools/techniques – clickmaps/user feedback/usability testing. The final result was an increase in 400% increase in conversions.

Lesson: simplicity; social-proof; role of video in conversions; clarity in design

Skype Boosts Homepage Conversion Rate with Radical Simplicity

Skype tested their homepage for 3 different variations. Key difference between the three versions was the level of details presented on the page. Guess which variation won the test? The simplest variation of all (see image above) saw an increase of 5% in downloads. The figure may not look like much of an improvement but when you consider that Skype may be getting tens of thousands of downloads each day, this increase becomes quite large and totally worth-it spending thousands of dollars on doing the A/B test. (But, hey, you don’t have too – after all, Visual Website Optimizer is free to signup).

Lesson: simplicity, simplicity and some more simplicity

Do you know of any other interesting A/B split testing case studies which I should add in the post? Perhaps you may want to take help of  A/B ideafox (a search engine for A/B split testing case studies).

Founder and Chairman of Wingify.

Comments (13)

Leave a Comment
  1. Thanks for the post. Great to see such a varied set of split testing case studies. For me, the overwhelming observation from these tests is “simpler is better”: eliminate distractions, reduce required input, decrease the number of steps. Where I see the greatest benefits of A/B testing is after the low-hanging fruit of simplification has been harvested, and you want to test the subtleties of marketing, such as word choice, color, page layout, and offers.

  2. Our own A/B tests confirmed that less is more. With fewer options on a page, user will behave more as expected. I guess this fits in with Steve Krug’s model of not making the visitor think.

  3. Prior to them lumping it in with Analytics, we had a lot of success with Google’s Website Optimiser tool.

    It’s a shame they decided to remove it as a standalone A/B testing tool, because it was actually pretty good.

Leave Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes : <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Contact Us / Login