Talk to a sales representative

+1 844-822-8378
or

Write to us

Check Your Email!
We've sent a message to
yourmail@domain.com
with an activation link in it.
Just click the link, and we'll take it from there.
Can't find the mail?
Check your spam, junk or secondary inboxes.
Still can't find it? Let us know at support@vwo.com

VWO BLOG

on Conversion Rate Optimization

Many times A/B testing is not limited to a single conversion goal. In fact, your test variations usually affect many different conversion goals on your site such as free trial signups, paid signups, newsletter subscription, etc. Measuring all these goals for a test is important because a variation may work brilliantly for one goal (e.g. it increases free signups) but may perform worse for your other goals (e.g. decreases your paid signups). If you don’t measure multiple goals for your A/B test, you are essentially flying blind in the dark and may end up making wrong decisions (based on a single conversion goal).

Case Study

A brilliant example of this is a recent A/B test by one of our customers: Guido Jansen (a Magento specialist). He runs Dutchento.org which is the official Dutch community for Magento CMS. The goal of his recent A/B test was to increase subscriptions for Dutchento’s newsletter and RSS subscription. The call-to-action for subscription is located in a box on all pages of the site. See the control version of subscription box below and note that there is no explicit incentive for a visitor to subscribe:

Guido tested a variation of this subscription box which included a title and some benefits in a bullet list. Here is his hypothesis for increasing subscriptions:

People don’t just subscribe to a newsletter or newsfeed for nothing, you should convince them it has added value above just visiting the website. So what I wanted to test is if adding convincing reasons to subscribe would increase the newsletter and newsfeed subscription rate. I measured the impact of the convincing reasons on clicks on both the newsletter and newsfeed links.

Here is how his variation (of subscription box) looks like:

Results

As expected, Guido saw a significant improvement of the newsletter click rate (+190.31%). However, newsfeed click rate decreased (-44.46%) which did surprise him (and us!). He expected that the convincing reasons would affect both positively, but apparently it had a negative effect on newsfeed clicks.

The reason why clicks on newsfeed decreased is not clear but we believe that the benefits in variation were so compelling that visitors chose to get the blog updates via email (where they will be sure to read them) rather than RSS reader (where they may miss them). A great way to get more insight into this would be to randomize the position of newsfeed / newsbrief to eliminate the positional effect of those links.

Testimonial

Guido used Visual Website Optimizer for A/B testing and here is what he had to say:

Visual Website Optimizer was very valuable [for testing]. It’s the easiest A/B and Multivariate testing tool I know. It’s great not to be dependent on the development department to create and run your tests.

Key Lesson

Whatever be the actual reason of decreased clicks on RSS feed, one key lesson is jumps out of the case study: always measure multiple conversion goals in your test. Relying on a single conversion goal hides a lot of valuable information from you. So, make sure you add multiple goals to your next A/B test.

Do you have more examples of using multiple goals in A/B test? What are your thoughts on this case study? Please let us know by leaving a comment below – will be happy to discuss!

Author

CEO of @Wingify by the day, startups, marketing and analytics enthusiast by the afternoon, and a nihilist philosopher/writer by the evening!

(2) Comments

Leave a Comment
  1. […] use my test as a case study on their blog. Read on to learn what I learned by using split testing: A/B testing with competing goals: newsletter CTR increased by 190%, but clicks on RSS feed….[disclaimer]I’m a happy customer of VWO, but I use several systems to analyze and test and I […]

  2. We are A/B testing our Magento information page. We are looking forward to get the results! Which one had the highest conversion rate? The long page or the short page.

Leave A Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes : <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Close