Proving Returns from A/B Testing – 6 Ways to Keep Your Boss Happy

To gain results from testing, you need to believe in it strongly. You cannot look at testing like other channels or tactics and ask,

“Okay…so, what’s the return from testing this month?”

Testing is a culture, a mindset of optimization. You MUST look at the bigger picture here (A/B testing could actually help you get a promotion!)

Sometimes you might end up running a series of unsuccessful or somewhat successful tests before you hit gold with a winning test. That test will be your jackpot. The one whose revenue boost will more than make up for the lost time and money you invested in testing the past few months.

Download Free: A/B Testing Guide

The problem here is, you may believe in testing until the end of time, but proving it as a viable investment source can be extremely difficult. When you have nothing much to show for a while, or when you’re spending on testing before you’ve started to gain from it — how do you justify this cost to your boss? Keep him happy with the results and let you continue testing?

Below, I’ve compiled a few ways for you to get the maximum return and justify your testing spend:

1. Let Go of ‘Test One Page Element at a Time’ Rule

I’ve been an advocate of this conventional CRO bite — ‘test one page element at a time’ — for far too long to refute it now. But there are times when it seems best to let it go in favor of pragmatism.

Single element changes often take much longer to achieve statistical confidence. Plus, every test you run will not be a winner. So when you are playing too safe and running only small tests, a lot of time may pass when you do not have much to show for it.

When you make multiple changes at a time, you might miss out on customer learning. But that’s okay. Some changes on the page might increase your conversion rate, and others may reduce it. Get over it! Sometimes it’s the overall positive effect that counts. Remember that!

It’s thus necessary that you break free from this conventional bite of testing wisdom and not be scared of making big changes.

If you spot multiple conversion leaks in a page that needs fixing, go ahead and make a new page that addresses them all; and then test it against the original page.

Small changes do have big impact sometimes, but those are a handful of cases. Most times, you will have to make more than one change to see the drastic difference in the way your visitors behave.

Start with best practices…

If you have a really leaky page from a conversion standpoint, don’t shy away from starting with best practices.

Yes, best practices do not work for everyone. And all of them might not work for you as well. But a lot of them should work for you and it’s a great way to add some quick fixes and have some good lifts to show off.

2. Sort Your Test Priority

Once you’ve completed your end of research and analysis, you’re likely to have tons of hypotheses all ready to put into action. Of course, you cannot try all the tests in one go. To show wins to your boss without much delay, prioritize which tests you can run first and without much friction. Gradually, move towards difficult tests once you’ve gained his confidence.

Wider Funnel’s PIE framework comes handy in deciding test priority. Make a table like the one given below, add hypothetical scores out of 10 for each factor (Potential, Importance, and Ease), find a PIE average for each test idea, and then decide:

While some big tests might need extensive assistance from your tech team, there might be others that need a rather daunting approval from the management. By following the PIE framework, way you will smoothly move forward with a sorted testing strategy and first focus your efforts on tests that combine high revenue potential and easy implementation.

Once you’re all pumped up with some good results, you can then stretch for other difficult tests.

3. Create Theme-Based Page-Level Tests

Theme-based tests are my personal favorite. They are a perfect example of hitting two birds with the same stone. You can change multiple things at a time and still get an actionable customer insight from the test. The only twist here is that the changes you make should be based on a particular theme.

A rehab facility chain, Tuscany, for example, tested their original landing page that focused on the extravagance and secluded location of their facility against a new version that emphasized on building trust in the mind of the prospects. This gave them a lift of 220%. Plus, they now understand that trust is a more important concern for prospects than a luxurious facility.

4. Using Test Insights to Up Your Overall Marketing Efforts (Including Offline Campaigns)

Experts often insist that you must look for customer insights in your test results. Many people do not understand why it is so important. So they ignore the reasons why their visitors behave in a certain way, why they buy/didn’t buy from their website. Missing out on these crucial insights mean — they use testing on its face value and will never realize the true potential/benefits of testing.

They fail to see that they can apply these customer learning to improve their overall marketing efforts, including offline communication. Continuing the same rehab facility example above, Tuscany applied the customer learning from their test and adopted the trust-focused approach on their other 300 websites. This gave them a 85% boost in paid search revenue across all 300 websites.

From their landing page copies to customer calls, Tuscany’s entire approach of presenting themselves transformed their business after that.

Download Free: A/B Testing Guide

5. Run Site-Wide Template Tests

Because of the wider impact of these tests, their rewards are also manifold. Even a small win on these template pages can give you a huge lift to rave about. Lemonfree.com conducted a site-wide test on their product template pages, which increased their revenue per visit by 19%.

Apart from the usual category/product page templates, header and navigation tests are some other common site-wide tests you can try. Site-wide tests are also a great solution for those with low traffic count as the cumulative traffic of all template pages should give you enough traffic to get conclusive test results quickly.

6. Stick to Evidence-based Hypotheses

Conducting random tests that are backed by no research/insights or data will only waste your time and money in the long run. You must collect quantitative and qualitative data about your customers as well as your website to formulate smart hypotheses that have a higher probability of hitting the jackpot.

This means you’ll need to find high opportunity pages in Google Analytics for your website. Often high traffic pages with a higher bounce rate, checkout flow pages, sign-up pages, et al are good starting points. Tests run on pages where you land your PPC traffic can also have a high-impact on your revenue.

Next is to understand your customers. Don’t assume that you know what they think. You don’t! User-testing, reading live chat transcripts, and conducting exit surveys, first-time buyer surveys are the most powerful (and quite cost-effective ways) to know “why” people behave in a certain way on your website. Why they buy/don’t buy from you. You can then use these insights for your hypotheses and these will now be an educated guess, rather than an absolute shot in the dark.

Few questions you can ask to understand visitor intent or customer hesitations on your website are:

• Is there anything holding you back from making a purchase right now?
• Do you have any questions that you can’t find answers to on our site?
• What brought you to our site today?
• Were you able to accomplish the task you came to do?

One mistake most companies fall victim to is that they treat testing as a one-off tactic. Companies that recognize conversion optimization as a process and ingrains constant testing in their culture are the ones that see real wins from testing.

What’s Your Take?

Are you stuck in an organization where you’re struggling to make testing a mainstay? What challenges do you face?
We would love to hear your thoughts on marketing@vwo.com !