VWO Logo
Like this post?
Read our in-depth guide to Conversion Rate Optimization
Share this post
5 Min Read

Seven A/B testing mistakes you need to stop making in 2013

Siddharth Deswal
I do marketing at VWO.

We’ve survived the annihilation predicted by the Mayans and made it into 2013. Ain’t that absolutely awesome? What isn’t so great is all those testing mistakes you’ve almost certainly made through 2012, mistakes (or bad practices) that were holding back your A/B testing and Conversion Rate Optimization efforts.

For your benefit, here’s a quick recap of the sub-optimal practices you need to let go of, to truly achieve the gains promised by A/B testing.

1) Not calculating your sample size before starting the test

Many marketing folks still don’t calculate the number of visitors needed to run a test before starting the test. As pointed out by Evan Miller in his post “How Not to Run an A/B Test“, you need to decide the required sample size before the test. This ensures that you don’t get bitten by the euphoria (or depression) bug when you see your first statistically significant result and save yourself some bad decision-making.

To illustrate my point, most successful A/B test reports look like the one below.

A/B test reportClick image to see larger version

Notice the fluctuation in the beginning when there wasn’t enough data. It’s quiet likely that on Nov 28, statistical significance was reported but that would lead to an incorrect decision being made.

To avoid this situation, head on over to our Split Testing Duration Calculator, plug-in your values and run your test for the duration suggested by the tool.

2) You listen to your boss or the HiPPO more than the data

Data has truly killed the HiPPO (Highest Paid Person’s Opinion) Star. While age and experience in a particular function do have their own merit, A/B testing is about letting the customer do the talking, without actually talking. So as a business, you need to develop a culture of testing which values real analytics over personal opinion brought on from years of experience.

Data will kill the HiPPO

Experience should guide “the correct way to do it” and not “the exact way to do it”.

3) Disregarding test results and going by what “looks good”

To most of our readers, this might be baffling. After all, why test if you don’t intend to push it live? Unfortunately, it’s a situation we observe far too often. A test is run, a winning variation is found but someone who holds some sway within the organization says it “doesn’t look good”. Lo and behold, the original, losing version of the page stays.

Please leave the money on the table, cos it just don't look nicePlease leave the money on the table, cos that variation just don’t look good.
Image credit: emdot @ Flicker

4) Always expecting large results from small changes

One of the most commonly heard pieces of advice about A/B testing is to go for the small wins, or make small changes that get you big wins. This image we created aptly explains the problem with overusing that bit of advice.

Small wins making you miss the larger picture?

Overly relying on “simple tweaks” instead of undertaking major design changes means you’ll just reach what is called the local maximum. That is, you can tweak all you want but the current design has reached its maximum conversion potential. However, you could achieve much larger increases in conversion rate if you tried a completely new design.

5) Sticking to plain vanilla A/B testing

You are missing out on a lot if you’re not running targeted, personalized tests. When you engage with a visitor, you’re essentially starting a conversation and there’s a different conversation happening with different customer segments. For example, the Smashing Magazine article “The Ultimate Guide to A/B Testing” written by Paras sends a lot of traffic our way. To continue the conversation, we ran a targeted test on the homepage that changed the headline and the sub-headline to closer match where they came from.

VWO Smashing Magazine test

The result? For the goal being “Visitor signs up for Trial Account”, the targeted messaging is currently out-performing Control by 98.42% with 96% chance to beat original.

6) Not having a well defined Conversion Rate Optimization strategy

We see that a lot of A/B tests are done on hunches and without any planning. Someone is using part of a website and realizes that a particular UI element could do with some tweaking so he/she comes up with a hypothesis, a goal and either thinks up a variation or asks others in the office for their inputs. Based on this, one or two variations are quickly created by the in-house designer and the test goes live.

While this approach may provide small increases in conversion, you’re not truly harnessing the power or CRO. A great Optimization Strategy (PDF) starts with gathering extensive feedback from all stakeholders, pouring those insights into appropriate steps in your sales funnel and testing each step methodically.

7) Not celebrating the way a true win should be celebrated

At the time of a major A/B testing win, have you felt your body, your soul, in fact your entire existence urge you to get up and do the Victory Dance? Well I don’t know about you, but I feel like doing this dance every time one of VWO tests produces a winner.

Here’s wishing you a great 2013 and loads of Victory Dances!

Follow the very interesting HackerNews discussion here.

More from VWO

5 marketing mistakes you MUST avoid for better conversion rates and sales

Editor’s note: this is a guest post by Jeremy Reeves, who is a freelance copywriter…

Read More
Paras Chopra

Paras Chopra

16 Min Read

VWO Was Colart’s First Choice As An Optimization Platform

We spoke with Patxi Gadanon, who works as a Senior Web Manager for one of…

Read More
Smriti Chopra

Smriti Chopra

8 Min Read

A Psychological Guide to Improving Conversions

According to Harry Levi Hollingworth, the man who brought psychology into the world of advertising,…

Read More
Ankit Oberoi

Ankit Oberoi

17 Min Read
Shanaz from VWO

Hi, I am Shanaz from the VWO Research Desk.

Join our community of 10,000+ Marketing, Product & UX Folks today & never miss the latest from the world of experience optimization.

A value for this field is required.

Thank you!

Check your inbox for the confirmation mail