This website works best with JavaScript enabled. Learn how to enable JavaScript.

VWO Blog

A Blog Around Everything Conversion Optimization And Beyond

Never miss an update from VWO

A value for this field is required.

Your Intuition is Wrong – A/B Testing Results that Surprised the Experts

One of the things I absolutely love about split testing is its inherent ability to humble opinions of even the most seasoned testers. Sometimes even the most well researched hypotheses fail.

This is the strongest reason why companies must test everything from offer copy to page design, instead of relying on gut instinct or personal preferences. In this post, I’ll share some case studies that either saw huge opinion disparity among the WhichTestWon community, or whose results absolutely shocked our editorial team and Testing Awards’ judges.

Social Proof Is Not the End-All-Be-All

Can you guess which of these two versions of an in-line form generated more opt-ins for a well-known Web design blog?

With social proofWithout social proof

As of today, 71% of the WhichTestWon community picked the version on the left – the one with social proof copy – as the winner. When I present this test to live audiences at conferences, usually 90-95% of attendees pick the social proof version.

However, they are all wrong. The variation at the left — without the subscriber count — saw 122% more newsletter opt-ins than its social proof counterpart. In a world where we are jaded by the user counts of Facebook, Twitter, etc…it seems that 14k visitors wasn’t compelling enough to get prospects to act.

Normally we see companies just add social proof without testing it because virtually every blog post and ‘social media expert’ has told them it will help conversions. Thankfully the team at this site tested this before implementation, or they would have missed out on a lot of opt-ins – more than double the opt-ins in fact.

Don’t get me wrong, there is a ton of value in social proof. For one, it is great at reducing a visitor’s anxiety and can give your brand a sense of authority. The trick is finding out where it is best to publish your social stats and what exactly is worth sharing. Should you post newsletter subscribers, Facebook likes, awards, or all of the above? Simply put: test it out. Never add something blindly; you don’t know how it will impact the bottom line.

Still not convinced? A VWO customer saw a similar result recently on its product page. Read the whole story here.

Icons May Be Trending, but they Might Hurt Conversions

Icons have been making a major comeback in web design. Overall, icons have been useful, especially when they are used as a replacement for bullets in stylized lists. The team at wanted to find out whether icons would be a useful navigational tool…the results surprised them.

Here are the two versions of the header they tested, one with icons and one without:

Without icons
With icons

The challenger variation included icons that represented different category pages on the site. The team believed that an increased focus on navigation with their most visited categories would increase interactions and sales. However, the version without the icons saw 21% more product purchases.

Why? We suspect that although the icons provided a sleek navigation pane, overall they likely added more clutter that confused the visitor.

Security Seal on a Lead Gen Form Test

The highest point of friction on any lead generation page is the form itself. You need to identify the optimal number of form fields, choose an intuitive design, and add visible privacy policies and/or security icons to reduce anxiety.

These are well-known best practices that all lead gen marketers understand… and that’s probably why 74% of the WhichTestWon community guessed the wrong winner for this A/B test.

Without trust seal
With trust seal

The variation without the TRUSTe logo got 12.6% more completed forms. Yes, the ‘submit’ button did shrink to accommodate the TRUSTe logo; but, we strongly suspect the primary cause for this lift has to do with the logo itself.

Trust seals can be essential to your conversion rate; the real trick is knowing where and when to place them.

In this particular circumstance, the TRUSTe logo was the wrong security seal at the wrong time. Visitors are used to seeing this seal, and others like it, directly in a shopping cart; not on a top funnel lead generation form. It’s quite likely that many of them suspected a payment transaction when they saw the trust seal here.

Instead of using a security seal, the team could have tested providing assurance by adding a simple fine-print text link to the privacy policy.

Remember, context is the key!

If Your Conversion Rates are Falling, Put on a Happy Face?

CRO specialists and designers love using faces! I get it; faces are the first thing the eye identifies when it looks at a web page. Numerous eye tracking studies support this claim.

However, sometimes a human face can be too much of a distraction. So, you need to test when you add faces to your page to make sure they aren’t competing with your important headlines and calls to actions (CTAs).

Here’s an example:

Without human face
With human face

The version without the image won 24% more form completions. This wasn’t a perfectly clean test. There were some slight alterations in copy but nothing too dramatic. To tell you the truth, I’m more of a fan of the story behind this test than the actual unexpected results. However, it’s not the first or the last test we’ve seen where removing a face increased conversions.

By the way: I am happy that the team used the photo of an actual employee rather than a stock photo. Models and stock photos tend to get even worse conversions than “real” people.

What’s perhaps most amazing is that at the time of this split test HubSpot was about to make it a mandatory practice to include a person’s image on each of their landing pages. On some level this makes sense, they had found that some of their pages performed better with people’s pictures. However, what’s true for some pages may not always be true for all pages. Luckily, this test casted a seed of doubt and the company changed their design mandate.

Before you create any new landing page or design policies, please test beforehand…you have no idea just how many conversions you could leave on the table.

Video Icons – Product-Centric or Person-Centric?

Here is another case that tests whether using a face is appropriate.

Person-centric video thumbnails
Product-centric video thumbnails

The version of this Autodesk product page that used faces got 50% fewer video clicks. Nothing else on this page changed except for the video preview image. I am not anti-faces on websites; I simply want you to test before you implement!

Needless to say, the testing team was surprised by the results. So they ran a user survey to try to figure it out. The responses showed that Autodesk’s prospective buyers were more interested in seeing how the product worked over individuals talking about the product.

This just comes down to a case of knowing your audience and that best practices are not one-size-fits-all!

In Summary

Leaders in the testing field have all been stumped by the unexpected results before, and will be stumped again. The trick is to understand what to do after your test goes counter to your hypothesis or flat-lines.

Your next steps may include evaluating a litany of things such as your hypothesis, technology, source traffic, device…the list goes on. You need to learn if the test itself was flawed – or if your understanding of what your visitors really want from the page was flawed. Either way, you’ve learned something valuable.

Remember testing is an evolving process, future iterations are borne from our successes and our failures.

Keep testing my friends! There are so many variables to consider while running a test, it is no wonder that we often see lifts or losses where we least expect it.

UX Case Studies (eBook) CTA

Comments (11)

Leave a Comment
  1. My favorite tests are the ones where the obvious (or hypothosized) winner actually performs worse. As testers, we learn far more about a customer group from the things we least expect than the “obvious” wins, or even strongly researched hypotheses that are wrong.

    Most recently I’ve learned that some customers actually convert higher when they are interrupted by an email popup. Another customer base decided they are more likely to checkout with an unnecessary step in the process.

  2. “As of today, 71% of the WhichTestWon community picked the version on the right – the one with social proof copy – as the winner”

    i think it’s the left image, not the right.

    As for those testing results, it won’t be much of a surprise if we understands that most of the time our users/audience likes a simpler content and a cleaner page which means less work for their brain to do or understands.

    I’ll definitely will use the approach on my next landing pages work.

  3. I have to say that I agree with Azizul Yusof. In particular, using faces is potentially problematic for many reasons. The many different, and often unconscious, feelings generated when we look at faces are more likely to have a negative impact on calls to action. Even in cases were the face is deemed attractive and inviting. Humans can be quite perverse sometimes! 🙂

  4. My rule of thumb regarding faces is to just NOT do it.

    Stock face images always feel like the lowest form of image possible.

    To me it says:

    1) Our product is so boring we just used this fake person instead.

    2) We are afraid of showing a real employee, here’s a model instead.

    3) Lack of artistic inspiration.

  5. @Abu Adham I tend to agree. It also rather depends on what you look like. If you resemble Colonel Saunders maybe it’s a good idea. If you resemble a serial killer, maybe not such a good idea 🙂 eg: Pat Flynn, yes. John Chow, NO!

  6. That’s really great to see those small things that have much more impact on the business. I completely agree with you by using those stats in subscription forms, security seals on lead generation forms, a real face on lead forms really have a great positive impact on accomplishing the purpose.

    I too have noticed many times, people are more expected to subscribe through forms which shows a high subscription numbers along with the form. This makes trust and also acts as a proof for the good quality factor of that service.

    A happy face inspires people to join or to know more about the campaign. And this way generating more leads. And in case of video marketing, the face of person on the video thumbnail makes it more realistic and their click rates are higher than videos with non-human thumbnails.

    It’s a great read Sunday. You have put something really valuable and must know for web marketers.
    Thanks for sharing it on

  7. I think we need to understand what social proofs can prove and not. There is a bigger story that will surprise the experts. This psychological phenomenon isn’t that satisfying at all, so I harmonized with the statement and the explanation that “Social Proof Is Not the End-All-Be-All” shared.

    Think about real things that will happen. Dig into more information, reaction or an authentic result that will persuade you in making a shrewd and efficient decision.

    I have shared this comment in the content syndication and social bookmarking and networking website for Internet marketers – where this post was found.

  8. @Sandeep Rao
    Sorry, but did you actually read this piece?
    The article clearly demonstrates the exact opposite of what you are saying. It points out cases where smiling faces, social proof and security seals have a NEGATIVE effect!
    Hence the headline: “Your Intuition is Wrong – A/B Testing Results that Surprised the Experts”

  9. @Mike
    But most of the times, I felt that a large number of social media make a great impact on new users. And that’s why I said it a compelling factor for new sign ups.

    Truly speaking, If I have to join some career coaching group, and I found that its Facebook fan page have even less than 100 fans, I would look for some other group for sure.

    These days, social presence make a positive sense of reputation and quality too.

Leave Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes : <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Contact Us / Login