VWO Logo
Like this post?
Read our in-depth guide to Conversion Rate Optimization
Share this post
5 Min Read

Seven A/B testing mistakes you need to stop making in 2013

Siddharth Deswal
I do marketing at VWO.

We’ve survived the annihilation predicted by the Mayans and made it into 2013. Ain’t that absolutely awesome? What isn’t so great is all those testing mistakes you’ve almost certainly made through 2012, mistakes (or bad practices) that were holding back your A/B testing and Conversion Rate Optimization efforts.

For your benefit, here’s a quick recap of the sub-optimal practices you need to let go of, to truly achieve the gains promised by A/B testing.

1) Not calculating your sample size before starting the test

Many marketing folks still don’t calculate the number of visitors needed to run a test before starting the test. As pointed out by Evan Miller in his post “How Not to Run an A/B Test“, you need to decide the required sample size before the test. This ensures that you don’t get bitten by the euphoria (or depression) bug when you see your first statistically significant result and save yourself some bad decision-making.

To illustrate my point, most successful A/B test reports look like the one below.

A/B test reportClick image to see larger version

Notice the fluctuation in the beginning when there wasn’t enough data. It’s quiet likely that on Nov 28, statistical significance was reported but that would lead to an incorrect decision being made.

To avoid this situation, head on over to our Split Testing Duration Calculator, plug-in your values and run your test for the duration suggested by the tool.

2) You listen to your boss or the HiPPO more than the data

Data has truly killed the HiPPO (Highest Paid Person’s Opinion) Star. While age and experience in a particular function do have their own merit, A/B testing is about letting the customer do the talking, without actually talking. So as a business, you need to develop a culture of testing which values real analytics over personal opinion brought on from years of experience.

Data will kill the HiPPO

Experience should guide “the correct way to do it” and not “the exact way to do it”.

3) Disregarding test results and going by what “looks good”

To most of our readers, this might be baffling. After all, why test if you don’t intend to push it live? Unfortunately, it’s a situation we observe far too often. A test is run, a winning variation is found but someone who holds some sway within the organization says it “doesn’t look good”. Lo and behold, the original, losing version of the page stays.

Please leave the money on the table, cos it just don't look nicePlease leave the money on the table, cos that variation just don’t look good.
Image credit: emdot @ Flicker

4) Always expecting large results from small changes

One of the most commonly heard pieces of advice about A/B testing is to go for the small wins, or make small changes that get you big wins. This image we created aptly explains the problem with overusing that bit of advice.

Small wins making you miss the larger picture?

Overly relying on “simple tweaks” instead of undertaking major design changes means you’ll just reach what is called the local maximum. That is, you can tweak all you want but the current design has reached its maximum conversion potential. However, you could achieve much larger increases in conversion rate if you tried a completely new design.

5) Sticking to plain vanilla A/B testing

You are missing out on a lot if you’re not running targeted, personalized tests. When you engage with a visitor, you’re essentially starting a conversation and there’s a different conversation happening with different customer segments. For example, the Smashing Magazine article “The Ultimate Guide to A/B Testing” written by Paras sends a lot of traffic our way. To continue the conversation, we ran a targeted test on the homepage that changed the headline and the sub-headline to closer match where they came from.

VWO Smashing Magazine test

The result? For the goal being “Visitor signs up for Trial Account”, the targeted messaging is currently out-performing Control by 98.42% with 96% chance to beat original.

6) Not having a well defined Conversion Rate Optimization strategy

We see that a lot of A/B tests are done on hunches and without any planning. Someone is using part of a website and realizes that a particular UI element could do with some tweaking so he/she comes up with a hypothesis, a goal and either thinks up a variation or asks others in the office for their inputs. Based on this, one or two variations are quickly created by the in-house designer and the test goes live.

While this approach may provide small increases in conversion, you’re not truly harnessing the power or CRO. A great Optimization Strategy (PDF) starts with gathering extensive feedback from all stakeholders, pouring those insights into appropriate steps in your sales funnel and testing each step methodically.

7) Not celebrating the way a true win should be celebrated

At the time of a major A/B testing win, have you felt your body, your soul, in fact your entire existence urge you to get up and do the Victory Dance? Well I don’t know about you, but I feel like doing this dance every time one of VWO tests produces a winner.

Here’s wishing you a great 2013 and loads of Victory Dances!

Follow the very interesting HackerNews discussion here.

More from VWO

Data-Driven Optimization: How The Moneyball Method Can Deliver Increased Revenues

Whether your current ROI is something to brag about or something to worry about, the…

Read More
Jon MacDonald

Jon MacDonald

5 Min Read

How To Optimize Conversion Rate On Website Templates

The huge number of website templates available today has made it easier than ever to…

Read More
Sujan Patel

Sujan Patel

7 Min Read

Eating Our Own Dogfood – How To Optimize For Revenue As A SaaS Business

It wouldn’t be an exaggeration to say that we at VWO are very passionate about…

Read More
Zoeb Khan

Zoeb Khan

7 Min Read
Shanaz from VWO

Hi, I am Shanaz from the VWO Research Desk.

Join our community of 10,000+ Marketing, Product & UX Folks today & never miss the latest from the world of experience optimization.

A value for this field is required.

Thank you!

Check your inbox for the confirmation mail