This website works best with JavaScript enabled. Learn how to enable JavaScript.

VWO Blog

A Blog Around Everything Conversion Optimization And Beyond

Never miss an update from VWO

A value for this field is required.

Unexpected A/B Test Results that Defy Best Practices

Compare Courses provides an exhaustive database of courses to make it easy for prospective students to achieve their professional goals by choosing the most appropriate course for their growth.


On their site, you can find, select and get started with education courses that are studied in Australia from all over the world. This includes both government and private institutes of the country, and expert advice is also available for students from all disciplines and varied interests.

The Business Need

The majority of prospective students are not sure about the most suitable options that are available for them. And yet there are 1500+ Australian institutions that are offering over 50,000 courses.

Compare Courses aims to provide necessary help to these students in figuring out the right course for them. There are primarily three lead generation CTAs that run throughout their website:

  1. Lead Enquiry Form
  2. Ask An Expert Column
  3. Request Brochure Button

All the three lead generation methods are equally valuable for Compare Courses and the main focus was to improve the overall conversion rate of the website.

Test Hypothesis

Probably because Bachelor of Accounting is one of the most widely sought courses, the focus of this case study was the landing page given here:

CompareCourses Control

As shown in the image above, you can see CTA for all three lead generation methods clearly pointed out.

Now while, all the three lead generation CTAs were equally important, the main hypothesis was made to increase the conversion rate through the lead enquiry form. Compare Courses team decided to switch the placement and design of the lead enquiry form to bring it above the fold, probably thinking that this will get better attention to the form, and their lead generation efforts for the lead enquiry form will finally pay off.

The new form was symmetrically designed to look better than the previous one. Its placement was above the fold, and the social proofs and a testimonial was also displayed next to the form, for better credibility.

The placement of the “Ask an Expert” form was also shifted to support the design symmetry of the page. And it now appeared at the same place where the lead enquiry form appeared in the Control Page.

The placement and design of the “Request a Brochure” method remained unchanged in both versions.

This is how the Variation Page looked like:

CompareCourses Variation

Since the variation page required significant code changes, Compare Courses ran a Split URL testing. All three goals of lead generation were tracked for the above page.

The Unexpected Results

As the result tracking began, our customer didn’t realize how unexpected A/B split testing can turn out to be sometimes. Even though Compare Courses have always been using Visual Website Optimizer for testing various elements on their website, this was one test where all their insights, best industry practices for A/B testing and all other logical reasoning failed to justify what their A/B test results revealed. Here’s what they found out:

  • Lead generation rate from Ask an Expert CTA on the page showed a percentage increase of 159.52% with statistical confidence of 99.99%.

  • Request Brochure recorded a percentage improvement of 13.3% in the lead generation rate with 99.99% chance to beat the original.

Now if you remember, there were no changes that were actually made for these two methods. Almost everything remained untouched except the little placement of the Ask an Expert form to keep up with design aesthetics of the web page.

Last but not least is the main change that formed the basis for the hypothesis, i.e the new design and placement of the lead enquiry form.

Considering better design, testimonial and social proofs that were added and bringing the Lead Enquiry form above the fold, it was unexpected to see that the Lead Enquiry goal performed much worse on the variation page, and revealed a 53.87% DECREASE in the lead generation rate.

As a result, the Original version of the page converted leads at 9.71% and the Variation converted at 9.52%. This might seem like a small difference, but due to the large number of visitors on the page, it meant a loss of 100+ leads from the variation design.

Here’s the comparison image of the two versions that were tested (click on image to see larger version):

[pinit url=”” image_url=”” description=”CompareCourses split tested a new variation page which included best practices, yet it generated lesser leads than the original.” float=”left”]
CompareCourses Comparison

This makes it clear that you should never take anything on face value or go by what just “feels” or “sounds” right. It’s high time that we all should leverage the power of A/B testing as one can never be 100% sure about the changes made until you test them.

Never forget that even the the most often believed-to-be “best practices” of the web can misguide you. This is why it is essential for you to check the hypothesis before you make the changes permanent on your landing pages.

Another interesting point that came into light because of this case study is that when there’s more than one CTA on the page, it’s always best to track the changes for all of them, even when you’ve made changes to improve the CTA for just one of those elements on the page.

For example, not tracking the overall conversion of the page and only considering the lead generation through their Lead Enquiry form could give inaccurate insights to Compare Courses about their overall lead generation rate.

Comments (10)

Leave a Comment
  1. Very interesting analysis, and a well done post. Just a quick question: I noticed that the page was related to Bachelor of Accounting, and that the first CTA was to download a brochure. The most important CTA, as far as I understood, was in the “Enquire about this course”, that likely has the same keywords as the search. Also, by the numbers that you mentioned, it was probably the one with the highest clicks. By moving it to the right, I think that many people did not fully see it. Did you try the opposite, to move the “Request Brochure” to the right and give more evidence to the “Enquire about this course?”

  2. I wouldn’t call the decrease in Send Enquiry lead generation rate unexpected considering the notion of banner blindness – mentioned by Jakob Nielsen and others. As Caetano Notari suggests, people probably “deliberately” overlook the form to the right judging it to be irrelevant advertising banners.

    I would advise CompareCourses to cut down on the graphics and overall visual clutter. The page is simply “trying too hard” with three in-our-face CTAs.

    From a quick look it is not clear that both Request Brochure and Send Enquiry are about the same thing – the course. The elements look like disjunct entities. This is due to a confusing hierarchy of elements, e.g. the green header is more prominent than the page header. Again, the Send Enquiry is trying too hard.

    Another issue is this: It is not clear what to expect to get when Requesting Brochure and Sending Equiry.

  3. I just don’t see how this could remotely be surprising. The form was moved away from the main column to the sidebar. How could that possibly be considered a best practice?

  4. Hi Caetano, although all the CTAs were equally important for Compare Courses, I’ll have to agree with you that visitors probably didn’t pay much attention to it, considering it to be a banner ad or something, like it has been pointed out by Adreas in the next comment. It would be interesting to see the response if the two CTAs (Request Brochure and Lead Enquiry form) are placed the other way round.

    Hey Andreas, those are very interesting insights that you bought up about banner blindness and the green header being more prominent than the page header.

    Still, I will have to disagree with you on reducing the visual clutter and the lack of clarity of both the CTAs requiring action about the same course. To me, it seems clear as there’s only one course that was covered on the page. But yes, maybe briefing about the kind of enquiries that can be submitted or what readers should expect in the brochure may help Compare Courses in getting a better conversion rate.

    Hi Jacob, here are a few reasons why I insisted on the variation following the best practices:

    1. The Lead Enquiry form was bought above the fold

    2. Social proofs and a testimonial was displayed next to the Lead Enquiry form for better credibility.

    3. The new form design was better and clearer than the previous one.

    4. Privacy policy starts with “We respect your privacy,” which was missing in the Control.

    What do you think?

  5. Hi Smriti, I agree with your comments, they make sense. As I don’t know anything about their business model, I will use some assumptions – I may be wrong.

    It is remarkably similar to my toughest job, a lead capturing page for a company that sold leads to other companies (a long story…). The clients wanted name, phone, email and brief info about potential customers, so the form evolved to a very specific one, just with enough info to capture them.

    Looking back at this page, it seems that they want to capture the same info, and sell that to universities. If that is the case, I would make these changes:
    1) I would try to make the “Enquire about this course” the main focus on the page. With all the changes that you proposed.
    2) I would get rid of the Request Brochure at this page, and move it to the confirmation page after the “Enquire about this course”
    3) “Got a question” below the fold seems fine

    But, if they receive ad revenues for placing the “Open Universities”, I would move it to the right, instead of putting on the confirmation page…

    It all depends upon the business model, maximizing revenues… The fun things that we are not allowed to discuss online…

    What do you think?

  6. That’s right, Caetano. A lot depends on the business model indeed. I’ll reach out to Compare Courses and suggest this as a follow up test, in case they’re interested.

    Switching the Request Brochure CTA to the confirmation page can improve their conversion rate, I believe. But I do know that all three CTAs are equally important for the company, so I’m not sure if they will be keen to try it out.

    I noticed that they do have a “Start Date” option at the centre of their page, which says that interested students should send an enquiry to know about the date. Maybe changing the color of the text to red and changing the text from “Please send us an Enquiry” to “Please fill the Enquiry Form on the Right” will help the form in getting the necessary attention to get better conversions?

    What do you say?

  7. I think this statement may be causing you a problem:

    “…I do know that all three CTAs are equally important for the company”.

    Usually multiple CTAs aren’t a good idea. And it’s pretty unlikely that all three CTAs are REALLY equally important. I suspect that the client just doesn’t know or can’t agree internally on what the prioritisation of CTAs should be.

    Have they really done analysis that shows them that each different CTA has the same actual value to them if taken? Or they’ve just discussed it in a meeting and couldn’t agree?

    might be an idea to bang some analytics in to see what the final value of each CTA is or just do some common sense modelling. then you can adjust the CTAs based on that/


  8. I agree,

    One can’t expect a form or button in a sidebar to do well. Even more so when there is a different, more prominent CTA in the main section.

    What surprises me about this test is that the “Request A Brochre” CTA went up with another CTA to distract from it.

    My only suggestion might be that the unattractiveness of a CTA in the sidebar focused users and made the choice between the two buttons easier. It might have a similar effect to how a decoy offer works – making the other option more attractive by comparison.

  9. @Ian, agree with you. Prioritizing CTAs is the key.

    @Marcus, yes that can be one reason. We all know that the F-shaped pattern often make us ignore the right-sidebar of the page. I know I do even when there are no banners. That’s just how we read (scan) things online.

Leave Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes : <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Contact Us / Login