Optimization v/s Validation – two distinct uses of A/B testing
It is a myth that A/B testing can only be used for optimization and you need to fully establish a process optimizing it. In other words, many people think that A/B testing is best meant for big guys who already have an established conversion funnel, and now want to squeeze an extra 0.01% out of it. As a natural consequence to this logic, it may appear that A/B testing cannot be used at initial stages of a project (or startup) because there is nothing to optimize yet.
Wrong. There are actually two distinct uses of A/B testing: a) optimization; b) validation.
How to use A/B testing for Optimization?
As mentioned in the opening paragraph, this is the activity with which A/B testing is usually associated. Typical scenario: you are running an advertising campaign where you collect leads. You get 10 leads for every 100 visits and you realize at this conversion rate (10%), this activity isn’t profitable. So, you set out to optimize (or increase) the conversion rate. You test headline, product image, form length and a bazzilion different page elements. In the end, you manage to increase the conversion rate to 15%. Well done!
As you can see, with this A/B test, the objective was clear: optimize conversion rate for the activity one is currently doing. There are several VWO case studies where the tool was used for optimization. Examples include increasing sales by 20% by testing page design and increasing sales by 6.3% by testing Buy Now button.
How to use A/B testing for Validation?
Unlike above where we answer what is the best way to do an activity, in this case A/B testing is used to answer which activity to do in the first place. Validation means coming up with different possibilities and testing which one works best. Typical scenario: you decide to publish a new whitepaper for gathering leads. However, you are unsure which whitepaper will be most interesting to your customers. To answer that question, you make 5 different landing pages for each whitepaper and setup an A/B test. After the test is over, you check which whitepaper garnered most interest (via pre-release signups or some other conversion metric). Notice that you aren’t optimizing the lead collection process here, rather you are testing which whitepaper will potentially collect most leads.
To give you some examples, there are several VWO case studies where validation was carried out. Here is how a startup validated its positioning and here is how the appropriate position for a promotional message was tested.
A/B testing isn’t just for big corporations who want to optimize the last drop out of their conversions. In fact, A/B testing is more useful at answering business questions. Want to know which product to launch next? Which feature to develop? Which new market to enter? A/B test it today!
What is your opinion about this? Did you ever use A/B test for validation, instead of optimization?