Paperstone is a London-based office supplies company. They sell over 20,000 products throughout the UK. And also manage in-house the development and optimization of their online store, www.paperstone.co.uk.
Test Background and Hypothesis
Almost everyone expects a bulk discount when ordering for office supplies. Paperstone got to know about this significant detail during their customer survey. And like every successful business, they took their customers’ words seriously.
The team was thus all set to launch the bulk discount deals on their website.
Often when businesses implement something that has been requested by customers, it turns out to be profitable. But just like AB testing the price, bulk discount deals are a little tricky. You might see an increase in the number of sales, but this need not necessarily mean an increase in your profit margin.
So, it is important that the number of items sold by you increase so dramatically that it covers up for the lost margins in giving discount on bulk purchases. And of course, it should provide additional profit too.
This was a crucial test for George Harris, the Director of Paperstone, and his team, as results could go either way. The hypothesis was that bulk discount deals on selected products should increase profit margin from the website.
This was the Control page for the test:
A site-wide test was implemented. Bulk offers were promoted on some of the popular products.
This is how the offer was displayed on the Variation page:
Surprisingly, the bulk discount offers didn’t make any difference to their conversions. The most obvious thing to expect when you give bulk discount deals is, increase in sales. If it turns out to be profitable or not is a different issue. Here, the offer was ignored by visitors completely.
It seemed that the new design created “banner blindness” for visitors. The positioning of the offer in the right sidebar only contributed more to this loophole. This was contrary to the team’s expectations who believed that the “bright yellow bulk-buy ribbon” would actually draw visitors’ attention.
After this initial failure, the team faced a lot of pressure to ditch the idea and move on to other AB tests. This was also quite a complicated test to run and track. After all, not all visitors who came to the site might be interested in buying products on which bulk discounts were available. So, the test should also ideally run longer to be absolutely confident about the results.
Despite these challenges, the team decided to give another shot to the same hypothesis. They used Visual Website Optimizer for revenue tracking of their test.
Given below is the new variation they used for their second test:
This new variation was tested against the same Control that is mentioned earlier in the post. Here is the comparison image that summarizes the two tests for you:
Each version received more than 10,000 visitors. The new Variation beat the Control and increased average order value by 18.94% for Paperstone. The increase in average order value for bulk discount offers was 5%. The revenue uplift was 16.85%. It is worth noting here that even though the bulk buy offer was available on selected products only. Still, there was a huge increase in the total average order value.
Next time you have a failed test, instead of discarding the entire hypothesis, make sure you analyze the execution for any loopholes as well.