How Paperstone Used A/B Testing To Achieve Higher Conversions

A challenger seeking to target the large, established brands in the UK office supplies market, Paperstone wanted to use its website to make buyers aware that it offered products at prices lower than what the larger players offered. Paperstone believed that doing so would increase sales and hence market share.

George and his team hypothesized that displaying their competitors’ higher prices on 5,000 product pages will increase clicks on ‘Add To Basket’ and their overall website conversion rate.

This is what control looked like:

The team decided to display competitor prices along with Paperstone’s prices on the relevant product pages of the web site. However, as the team considered this change, concerns arose about the possible impact on visitors/ buyers:

1. Would displaying competitors’ prices drive potential customers to competitors, if the buyer was not previously aware that our competitors sold the product or if the price difference was only small?
2. Competitor prices were not available for all products, thus, would Paperstone be perceived as being more expensive on product pages that did not have competitor prices?
3. Would adding competitor prices distract users?
4. Would the test results be statistically significant?

Finally, the team decided to set up the test so that visitors became a part of the test only if they visited one of the 5,000 product pages (out of 18,000) that had competitor price data.

This is what variation 1 looked like:

The test was run for over 12,000 unique visitors (50/50 split between control and variation). The test goals were to measure “add to basket” and conversions to sales. On both counts, the variation recorded a poorer performance than the control. However, the results were statistically inconclusive.

A second round of tests was run using another variation in which the following changes were made:

1. Competitor prices were moved below “add to basket”. This was to test if moving the “add to basket” button down in Variation one had a negative impact.
2. Put the Paperstone price next to the competitor prices and highlighted the savings. This was to make it easier for users to see the savings.
3. To eliminate the possibility of “banner blindness”, the design was made more “corporate” looking and the prices were shown under the header of “Price Comparison”.

This is what variation 2 looked like:

Using VWO, Variation 2 was tested for over 12,100 unique visitors (50/50 split between control and variation).

The second test was conclusive for an increase of 10.67% in conversion rate (at a 95% confidence) but inconclusive for clicks on “add to basket”.