Debunking seven common excuses for not doing A/B testing
Most likely, A/B testing isn’t a must do activity for you. After all, it requires spending some time thinking what to test and then some more time is required for coming up with interesting variations. (Luckily, VWO has removed all the technical and integration hassles so you save time there). All in all, it requires time and effort investment, whose value and returns may not be obvious to you. Fully agree with you on that front.
However, for most people, the bottleneck is creating their first A/B test. The technique sounds hyped up, but it really isn’t and there is proof of even the first, really simple test yielding extremely positive results. CityCliq, a service that helps local businesses have presence online, created their first test on VWO and saw (statistically significant) 90% increase in conversions. Point to note is that it was CityCliq’s first test, they tested a small headline change, saw results and they are now hooked to A/B testing.
Even when the value of A/B test is strikingly clear to me and companies like CityCliq, AquaSoft and MedaliaArt, there are many who somehow still manage to come up with excuses for not doing A/B tests. One by one, I tackle all excuses and try to convince why you should create your first A/B test right now! Following is the list of common excuses and my perfectly justified counter-excuses:
A/B testing takes too much time to setup (integration and technicalities)
A/B testing takes a lot of effort in coming up with different variations
While this is true (and entirely justified) for A/B tests where entire design is tested, this is certainly false wherein you are testing headlines, call-to-action and other small but significant elements on your website. With WYSIWYG editor and, making variations is as easy as working on Microsoft Word (now some may debate even this but I think Word is pretty easy to use).
There are no guarantees that I will get results with A/B testing
True, there are no guarantees. But then there are no guarantees for anything worth its salt. Clichés apart, unless you act on something you shouldn’t expect your sales and conversions to increase automatically. A/B testing is no magic, to be sure. You cannot simply signup for VWO and expect sales to rise. However, if you aren’t doing A/B testing (or other doing business/product improvements) you can definitely expect sales to stay flat.
A/B testing will hurt my existing conversion rates
No it won’t. Most sophisticated A/B testing tools, including VWO, have options to switch off or disable variations that aren’t performing well. So, you don’t really have to worry that your existing conversion rates will take a huge hit in case variations are bad. You can setup a test, sit back, relax and go have a hot cup of coffe.
I know the truth (or, in other words, I already know what will work on my website)
No, you don’t. Trust me, you don’t. There are lots examples of A/B testing where the winning variation was an underdog. Except for systematic testing, you have no way for knowing what will work and what won’t. Of course, if you really know the truth, try doing A/B testing of your existing design against your hunch-based improved design and let your boss, clients, colleagues know how you always knew yours is better. (Warning: this will either shatter your status as an expert or take it to the next level!)
It requires a lot of traffic
Yes and no. The significance of your results depend on a lot of factors, and not just only on amount of traffic. You can play around with this A/B testing traffic estimation calculator and actually see what amount of traffic do you really need for your test. In some cases even a traffic of 300-500 visitors may be good enough. (In other cases though, 10k visitors may not be enough). But if you are testing for dramatic changes on the site, you can expect to get statistically significant results with moderate amount of traffic.
I am not ready for it yet
Why? Have you ever thought when would you actually be ready? When you roll out the product? When you start getting a million visitors each month? When your customer base doubles? Frankly, A/B testing should be done at all the stages of evolution of a website – from startup to establishment. In fact, for startups A/B testing is even more important because in initial days there are so many questions to be answered related to business, market, product, etc. A/B testing helps you find the answers you need. (Like it did for CityCliq).
The key lesson here is that if you have been procrastinating A/B testing for long, you should do it right now. If you have any excuses for not doing A/B testing that I haven’t covered, leave a comment on this post. I will most definitely come up with a counter-excuse why you should be doing A/B tests all the time 🙂