This one A/B testing case study will leave you smarter and wiser
PriceCharting.com specializes in offering current and historic prices of new and rare video games and consoles such as Atari 2600 and Super Nintendo. Basically in exchange for a small fee, it ensures you don’t overpay when buying a video game.
On its product pages, Pricecharting has a call-to-action (CTA) that says ‘Download’. The accompanying text tells you the fee you need to pay to get the price list. For example, the product page for ‘Sega Genesis’ console says ‘Sega Genesis Price List for $2’. This is how it looks.
JJ Hendricks, the owner of PriceCharting.com, felt that the existing CTA button looked too much like an advertisement.
“The ‘Download’ button and the price made buttons for our price guide service look too much like banner ads. We felt users probably had ad blindness and didn’t click it.”
This is how the Variation looked.
A/B test results
Just a week into the test, the variation emerged as the clear winner with the almost perfect 99.9% statistical significance. Changing the CTA text from ‘Download’ to ‘Price Guide’ increased the clickthroughs by a jaw-dropping 620.9%!
1) Greater relevance
Once you hit the ‘Price Guide’ button, you are taken to a page where you can customize your own video game price guide. According to Hendricks, ‘Download’ was not a very compelling CTA for his website users as they were looking for the Price Guide.
“When people view this (product) page, they are looking for the price guide for a particular type of video game.”
So you see? Just by making the text more relevant to his users, Hendricks reaped in a 620.9% increase in clickthroughs! ‘Price Guide’ was a better message and conversation match for the users as they were looking for the prices of the old video games. See how a fitness chain similarly increased its conversions by making its CTA text more relevant in point number 4 here.
2) More specific
‘Price Guide’ has a very specific meaning for the video game enthusiasts here. As opposed to that, ‘Download’ is a little vague and confusing. One doesn’t really know what will be downloaded after they click on the button. On the other hand, ‘Price Guide’ gives a clear, well-defined direction. It focuses on the goal of the page and fastens up the decision-making process of the potential customers.
3) We read from left to right
Our eyes happen to track text from left to right. The Control text, when read from left to right, reads:
Download Sega Genesis Price List for $2
The Variation text, on the other hand, reads:
Download Sega Genesis Price Guide
The first version starts with ‘Download’, but visitors are likely to think “Download what?” The second message starts with what is on offer and then finishes by telling visitors to click on the button.
Hendrick’s hunch had been proven right and he didn’t waste any time in implementing the changes on his website. But that’s not the end of the story. Like a true brilliant marketer, he didn’t stop testing just there. His curiosity had just started taking shape.
The second test
He ran a second test to figure out how price sensitive were his customers. It was a preliminary test to see how the customers behaved before he started a series of tests on the actual prices. So he cloned the CTA test and tracked the same goal (clickthroughs). On the previous variation (which was also the winner), he added a ‘Starts at $4’ next to the ‘Price Guide’ CTA. This is how it looked.
He set this one as the Control and tested two more variations against it. One with starting price of $2 and another with $1.
*This was an internal test and the new prices were not reflected on the checkout page.
The results of the second test
Though there wasn’t a clear winner, the $4 version did much better than the $2 and $1 variations with 91% statistical significance. However, the conversion rate of the $4 version was still less than the conversion rate of the winner in the previous test (0.70% compared to 1.03%). Hendricks did not add the ‘Starts at $4’ text on his product pages, but he got an invaluable insight that will hugely help him while deciding on his future prices.
“We will probably try some higher priced products on the landing page in the future to see if it will increase revenue.”
The fact that $4 value did better than the $2 and $1 versions means the visitors are agreeable to paying a higher price. Hence, Hendricks can safely test higher prices.
Why it worked
If one goes by the results of the preliminary test, it seems users value the products of PriceCharting.com higher than what was anticipated by Hendricks or they are indifferent to the price. In either case, it’s a win-win for Hendricks who has the required impetus to test the prices now.
One of our clients, Six Packs Abs Exercises, was also pleasantly surprised when it A/B tested its prices. An increase in price didn’t change the demand, bringing them more revenue.
Your key takeaways
1) Take it to the next level
You have got brilliant results in a test? Awesome. Do a little self-congratulatory gig, soak in the happiness and implement the changes on your website. And then live happily ever-after?
Start with the next test right away. Don’t stop. If we constantly try to make optimum utilization of our time and read all those self-help stuff to become more efficient, why not constantly try to optimize our websites?
2) Run a dummy test before testing high-impact elements
Pricing directly impacts your revenue. You are reluctant of A/B testing the price as you are scared it might affect your revenue. No worries. Run a dummy test or an internal test as Hendricks likes to call it. But do this only if you can carry it off.
A huge difference in prices on the product and checkout page might confuse the users. In that case, you could read this blog post to learn more about A/B testing prices.
Did you get it right?
Were you able to guess which version was the winner? What made you choose a particular option? Share your thoughts in the comments.