Talk to a sales representative

+1 844-822-8378
or

Write to us

A/B test case study: how two magical words increased conversion rate by 28%

Posted in A/B Split Testing, Case Studies on

When we saw the results Soocial had got from their latest A/B test, we were astonished! They added just two words next to the Sign up button and the conversions shot up by 28%. If we say the phrase was one of these: “Sign up for Free”, “It’s Free” or “Free Signup”, can you guess which one did the trick? That is precisely the beauty of A/B testing, you can never guess what works – they only way out is to actually test it. This case study has direct inputs from Soocial CEO, Stefan Fountain. He explains what they tested, why they tested and what others can learn from the results.

Background

Soocial is an online address book that helps you keep your phone, computer and online services contacts sane. The app syncs, merges and provides backups so that you are never stranded without the contact details you need. It works with over 500 phone models, webmail (like Gmail, Hotmail and Yahoo) and your Mac and Outlook. Soocial chose to use Visual Website Optimizer for their testing needs. Their first test was to increase the click-through rate on the homepage to the signup form.

What was tested

Their homepage features a large Sign up button. They wanted to test the age old principle of adding “free” to the call-to-action trigger. They tested a number of variations with different buttons including the word free, different colors and adding text next to the button. Part of this test was to see if correctness would trump brevity. For example one of the tested items was “Free up to 250 contacts” and another was “It’s free”. The former being technically more correct and the second being shorter but less “correct”.

Results

As you can read above, the changes on the page were extremely minor and, on the surface of it, look quite trivial. There is no reason why “It’s free” should work better than “Sign up for free”. Yet, it did! Out of several combinations, see the screenshot of the original version (control) and the winning variation.

Control: 14.5% conversion rate

Variation: 18.6% conversion rate

The only difference between winning variation and the original design is presence of “It’s free!” along side. And those two words increased the conversions by 28% (from 14.5% to 18.6%). This result was statistically significant and surely indicates that playing with the homepage really paid well for Soocial.

Why “It’s Free” Worked

When we asked Soocial why they thought the winning variation worked, this is what they had to say:

Being the nerds that we are creating Soocial, we thought that the most “correct” version would get the highest conversion. Of course the reason to use VWO is to confirm or deny our hypotheses and we will be testing a lot more variations in the coming weeks. Our hypothesis on the winning combination is that it doesn’t require any thought what-so-ever from users that it’s a low risk solution.

So, they believed that projecting the service as a low risk one did the wonder. We followed up the question asking what lessons can be derived from the test. Here is what they had to say:

To be honest we aren’t sure yet [of what impact this makes to the actual signups]. We wonder how these tests measure up to the goal funnels in Google Analytics and compared to actual conversions into paying customers. It could be that we get more users to signup and lose them later in the process. That is of course still valuable because it means we can have more call to actions to convert to a paying customer and test the conversion there.

For their next test, we recommend them to use VWO’s multiple goals functionality or VWO’s integration with Google Analytics to track the effect of variations throughout the funnel.

How valuable was Visual Website Optimizer for the A/B Test?

Here is what Stefan Fountain, CEO of Soocial, thinks about Visual Website Optimizer:

Invaluable. The ease and speed of setting up the test is brilliant and we can wait to start seeing the results on pages behind our login [a new VWO feature] to test on converting users to premium accounts. It will be very interesting to see the test results merely by changing the wording, graphics and positioning.

Key Takeaways

Two words can probably increase conversions for you as well. But the problem is that you don’t know which two words will work. Only an A/B test can answer that!

Comments (42)

  1. An interesting case study.

    I have a little qualm with this particular in that essentially it has been used to show that misleading the customer, however slightly, was more effective than not doing so.

    It may not be a huge deal but to me part of CRO is helping and assisting the customer towards the conversion goal rather than tricking them towards it. In this case it’s barely an issue for most but I bet some would be put out when they discover that they have to pay because they have 250+ contacts.

    Just a thought.

    Reply
  2. Well, right in your first paragraph when you said “can you guess which one did the trick?”, I knew it was “It’s free”.

    It’s easy to understand : “Signup for free” or “free signup” implies something like “it’s free to get in, but then you’ll have to pay for the product to be of any use”. Where as “It’s free” means “It’s free, you won’t have to pay anything”

    Reply
  3. Matthew says:

    What was the sample size?

    Was the difference in conversion rates statistically significant? at what confidence level?
    :)

    Reply
    1. Paras Chopra says:

      Matthew, the difference in conversion rate was significant at 95% confidence level. A lot of our customers choose not to reveal sample size and traffic.

      Reply
  4. Pingback: How two magical words increased signups by 28%

  5. What an excellent case study, it goes to show how it’s the small things that count! What I’d like to know is whether the main aim was to just get people to sign up, because then the goal was achieved, but in terms of long term results, a lot of the created accounts will probably be inactive, a lot of people just sign up for the sake of it when “free” is mentioned..

    Reply
  6. The key lesson here, which I often repeat on my blog is split test and observe the difference.

    It’s interesting how a simple change of words made all the difference with people signing up, but it would also be interesting to observe which set of customers purchases more also. Are the ‘it’s free’ customers expecting a free ride, where as the ‘sign up free’ people are expecting an upsell…

    If you were my client, that is what I would want you to test next. There are also many other variables that could come into play such as day of the week and where the advertising took place.

    But, kudos on a good mini case study on this split test.

    Reply
  7. Aren’t those three words? (not a native speaker btw)

    Reply
  8. Pingback: A/B test case study: how two magical words increased conversion rate by 28% « I love split testing – Visual Website Optimizer Blog : Popular Links : eConsultant

  9. Funny how the simple things bring the best results. Interesting article, keep them coming :)

    Reply
  10. Pingback: iPad Links: Wednesday, May 27, 2010 « Mike Cane's iPad Test

  11. Pingback: iPad Links: Wednesday, May 26, 2010 « Mike Cane's iPad Test

  12. I think what would give more positive result is if you specify how many records are actually free. Free often still means you can sign up for free but you need to pay to use services, and many users are aware of this trap.

    Reply
  13. Of course one can increase the number of sign-ups by lying – but do you really want to do this?
    You would act like a person with dubious moral standards and you have potential legal problems at hand.
    The art is to increase your sign-ups without being dishonest, not increasing the number of sign-ups by promising stuff you will not deliver.

    Reply
  14. Pingback: The Ultimate Guide To A/B Testing - Smashing Magazine

  15. Pingback: The Ultimate Guide To A/B Testing | Web Design Cool

  16. Pingback: The Ultimate Guide To A/B Testing | 

  17. Pingback: TG Designer » The Ultimate Guide To A/B Testing

  18. Pingback: The Ultimate Guide To A/B Testing « FED视野

  19. Pingback: The Ultimate Guide To A/B Testing | i know idea

  20. Pingback: 30 Multivariate & A/B Split Testing Tools, Tutorials & Resources | SEOptimise

  21. Pingback: PlayTheBalls - Agentie interactiva » Blog Archive » Resurse despre publicitate şi marketing online #4

  22. The FREE word always comes in handy. It’s really what attracts people to sign in and join what we offer.

    Reply
  23. Excellent point Greg! It does not get much more simple and easy to understand than “It’s free”. It’s interesting to see how far simplicity can go in increasing conversion rates!

    Reply
  24. What’s the p-value of your test?

    Reply
  25. Pingback: You should marry me…now! What to tweak and test in your call to action – Constant Responses

  26. Pingback: Making The Call: Using A/B Testing To Build A Better Website.

  27. Pingback: Call To Action - 16 tips och resurser hur du leder dina besökare rätt | Mikael Uusitalo

  28. Pingback: Making a business case for conversion testing | It's Digital Marketing

  29. Pingback: 3 Low Hanging A/B Testing Opportunities To Increase Conversion Rate

  30. Pingback: The Anatomy Of A Compelling Call To Action Button

  31. Pingback: 8 Ways your Landing Page Design is Sabotaging your Click-Thru Rate | Unbounce

  32. Pingback: Designing Websites to Increase Traffic and Conversion | Bitmag

  33. Pingback: A/B Testing Explained by Kalexiko Web Design

  34. Pingback: 50 Practical Examples for Exploding Conversion Rates

  35. It’s fairly obvious why adding the phrase “It’s free” helped improve conversion rates – the visitor is excited that the service/product/etc is free. However, they may become frustrated if they learn they have to pay somewhere along the line.

    Reply
  36. Its works…we change our call to action button same as here…then the conversion increase 25%

    Reply
  37. Pingback: UX Designer

  38. Pingback: The Call To Action In Emails | Scott Bratcher

  39. Pingback: La guida per condurre un Test A/B pubblicata da Smashing Magazine.

  40. Pingback: The Ultimate Landing Page Resource List

  41. Pingback: Generate more income when you sell more Things: Probably the most Properly Curated Listing of Landing Page Resources

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>