Unexpected A/B Test Results that Defy Best Practices
Compare Courses provides an exhaustive database of courses to make it easy for prospective students to achieve their professional goals by choosing the most appropriate course for their growth.
On their site, you can find, select and get started with education courses that are studied in Australia from all over the world. This includes both government and private institutes of the country, and expert advice is also available for students from all disciplines and varied interests.
The Business Need
The majority of prospective students are not sure about the most suitable options that are available for them. And yet there are 1500+ Australian institutions that are offering over 50,000 courses.
Compare Courses aims to provide necessary help to these students in figuring out the right course for them. There are primarily three lead generation CTAs that run throughout their website:
- Lead Enquiry Form
- Ask An Expert Column
- Request Brochure Button
All the three lead generation methods are equally valuable for Compare Courses and the main focus was to improve the overall conversion rate of the website.
Probably because Bachelor of Accounting is one of the most widely sought courses, the focus of this case study was the landing page given here:
As shown in the image above, you can see CTA for all three lead generation methods clearly pointed out.
Now while, all the three lead generation CTAs were equally important, the main hypothesis was made to increase the conversion rate through the lead enquiry form. Compare Courses team decided to switch the placement and design of the lead enquiry form to bring it above the fold, probably thinking that this will get better attention to the form, and their lead generation efforts for the lead enquiry form will finally pay off.
The new form was symmetrically designed to look better than the previous one. Its placement was above the fold, and the social proofs and a testimonial was also displayed next to the form, for better credibility.
The placement of the “Ask an Expert” form was also shifted to support the design symmetry of the page. And it now appeared at the same place where the lead enquiry form appeared in the Control Page.
The placement and design of the “Request a Brochure” method remained unchanged in both versions.
This is how the Variation Page looked like:
Since the variation page required significant code changes, Compare Courses ran a Split URL testing. All three goals of lead generation were tracked for the above page.
The Unexpected Results
As the result tracking began, our customer didn’t realize how unexpected A/B split testing can turn out to be sometimes. Even though Compare Courses have always been using Visual Website Optimizer for testing various elements on their website, this was one test where all their insights, best industry practices for A/B testing and all other logical reasoning failed to justify what their A/B test results revealed. Here’s what they found out:
Lead generation rate from Ask an Expert CTA on the page showed a percentage increase of 159.52% with statistical confidence of 99.99%.
Request Brochure recorded a percentage improvement of 13.3% in the lead generation rate with 99.99% chance to beat the original.
Now if you remember, there were no changes that were actually made for these two methods. Almost everything remained untouched except the little placement of the Ask an Expert form to keep up with design aesthetics of the web page.
Last but not least is the main change that formed the basis for the hypothesis, i.e the new design and placement of the lead enquiry form.
Considering better design, testimonial and social proofs that were added and bringing the Lead Enquiry form above the fold, it was unexpected to see that the Lead Enquiry goal performed much worse on the variation page, and revealed a 53.87% DECREASE in the lead generation rate.
As a result, the Original version of the page converted leads at 9.71% and the Variation converted at 9.52%. This might seem like a small difference, but due to the large number of visitors on the page, it meant a loss of 100+ leads from the variation design.
Here’s the comparison image of the two versions that were tested (click on image to see larger version):
[pinit url=”https://vwo.com/blog/unexpected-ab-test-results/” image_url=”https://vwocontent.s3.amazonaws.com/images/case-studies/CompareCourses_Comparison.png” description=”CompareCourses split tested a new variation page which included best practices, yet it generated lesser leads than the original.” float=”left”]
This makes it clear that you should never take anything on face value or go by what just “feels” or “sounds” right. It’s high time that we all should leverage the power of A/B testing as one can never be 100% sure about the changes made until you test them.
Never forget that even the the most often believed-to-be “best practices” of the web can misguide you. This is why it is essential for you to check the hypothesis before you make the changes permanent on your landing pages.
Another interesting point that came into light because of this case study is that when there’s more than one CTA on the page, it’s always best to track the changes for all of them, even when you’ve made changes to improve the CTA for just one of those elements on the page.
For example, not tracking the overall conversion of the page and only considering the lead generation through their Lead Enquiry form could give inaccurate insights to Compare Courses about their overall lead generation rate.