BLOG

on Conversion Rate Optimization

How RuneScape Leveled Up Revenue Through Process-Driven CRO

The following is a case study about how RuneScape followed a structured conversion optimization (CRO) program to increase revenue on its website.

About RuneScape

RuneScape is a fantasy massively multiplayer online role-playing game (MMORPG). It was developed by Jagex and launched in January 2001.

The popularity of the game is enormous. RuneScape has welcomed over 250 million players to its world since its release. More than 2 million users play every month, and millions more watch avidly through social channels.

RuneScape has consistently strived to deliver a great experience to its users—not just limited to the game but also on its website. After all, it’s the website where users find forums and game guides, and buy in-game items.

The CRO Team

Rob Marfleet, UX Specialist at Jagex, takes care of User Experience and CRO across the payment flow on the website (the payment gateway and its preceding pages). Dave Parrott, Payments Services Director at Jagex, and Nastassja Gilmartin, Payments Manager at Jagex, help Rob in identifying testing opportunities and analyzing test results.

Rob Marfleet, UX Specialist at Jagex, takes care of User Experience and CRO across the payment flow on the website.

Rob works with teams of designers and developers that help facilitate implementation of winning test variants on the RuneScape website.

Additionally, Rob takes help from Disha Ahuja, Client Success Manager at VWO, to utilize the VWO platform to its full potential.

About the Case

About 50% of users on the RuneScape website arrive as direct traffic. The other half of the traffic consists of users from referrals, social media, and email marketing campaigns.

Rob adds, “This is mainly down to RuneScape enjoying a very loyal user base, with many players having played for several years.”

The CRO team aims to optimize high-potential pages, that is, pages that are closest to the payment gateway and require minimum effort in optimization. The Treasure Hunter page on the website is one such high-potential page that the team chose to optimize.

The Treasure Hunter page lets users buy keys to unlock treasure chests in the game. The treasure chests contain items that can be used within RuneScape.

Rob explains, “Treasure Hunter activity is an optional mini-game within RuneScapekeys are earned through play, but can also be gathered in bundles that are purchasable on the site.

This is how the original page looked like:

RuneScape Treasure Hunter control page for A/B TestOn clicking Continue on the Treasure Hunter page, users are directed to a Payment page where they can choose from multiple treasure chest packages.

RuneScape payment page
Payment page

The RuneScape CRO team thoroughly analyzed the Treasure Hunter page and identified optimization opportunities. Next, the team used VWO to capitalize on these opportunities.

Optimization Process

The CRO team followed the following process to improve conversions on the RuneScape website:

  • Setting a Goal
  • Finding Opportunities for Optimization
  • Creating Hypothesis
  • Developing Variation
  • Analyzing Test Results

Setting a Goal

The goal of the optimization campaign was to grow revenue by increasing the number of purchases.

Finding Opportunities for Optimization

The team at RuneScape studied a heatmap of the Treasure Hunter page. The heatmap showed that a significant number of users were clicking the Get Keys section on the page—a section which was not clickable. Users perhaps either wanted a direct access to the keys or wanted to search for further information.

Heatmap of RUneScape original page before A/B test
Heatmap of the original page

Next, the team watched visitor recording sessions on the page and observed that a lot of visitors on the Payment page returned to the Treasure Hunter page. The team realized that the Treasure Hunter page probably did not offer sufficient information about the treasure chest packages to users.

Creating Hypothesis

The team hypothesized that providing details about treasure chest packages on the Treasure Hunter page will lead to greater conversions on the Payment page.

Developing Variation

Based on the hypothesis, the team created a variation of the Treasure Hunter page. The variation included a new section highlighting four treasure chest packages. Here’s how it looked:

RuneScape Treasure Hunter variation page

An A/B test was run to find the better performing version between the original page and the variation.

Analyzing Test Results

The test ran for a month from August 15–September 13, 2016. The variation outperformed the control and increased the number of purchases by almost 10 percent.

RuneScape A/B Test analysis - Report
Test result report on VWO

Rob shares his learning from the A/B test:

I think one of the more important aspects to take note of here is that the page variation actually resulted in less traffic to the payment page, but increased the amount of purchases made. Effectively, we can say pretty confidently that by giving the users package information upfront, we created higher quality traffic to the next stage, simply through transparency, and informed the user before going forwardusers who went to the purchase page already knew what they were after.

This is incredibly useful when considering other areas of the payment flowif the effect can be replicated, it can potentially translate to more wins.

Next Steps

The CRO team did not stop after it found success with the A/B test. The team felt that the variation can be optimized even further.

The team realized that the offer of four treasure chest packages can possibly leave the users spoiled for choice. The team hypothesized that recommending one of the packages to users will help them choose better and, consequently, increase conversions.

Based on this hypothesis, the following variation was created:

RuneScape follow-up A/B Test variation

The variation featured a Recommended package. This variation was pitted against the winning page from the first A/B test.

The variation won and further increased the number of purchases by almost 6%.

Experience Using VWO

Rob shares, “As a hands-on user of VWO, I’ve personally experienced how quickly it allows prototyping and testing of new ideas, features and content. The ability to push changes, without having to involve multiple teams to relaunch areas of the site can’t be praised highly enough, and the ability to reverse those same changes instantaneously is equally as useful. It’s allowed me to run a number of campaigns straight away that would normally have to be scheduled further down the line, at a more opportune moment, and that’s pretty invaluable.

Using the actual software is very straightforward and easy to understand—campaigns can be built in a short period of time, and having Disha available any time to help determine the best testing practices has definitely helped me find wins—she’s super friendly and eager to help, and I’ve already implemented several testing campaigns that have been borne out of collaboration between her and myself, one of which, is in the process of being fully implemented on the site.”

What Do You Think?

Do you have any recommendations on how RuneScape can further improve user experience and conversions on its website? Did you get any conversion optimization ideas for your own online enterprise? Tell us using the comments section below.

Nitin is a traveler, a cinephile, and a webaholic. (He just can’t get enough of cat videos!)
Professionally, Nitin is a marketer at VWO, who loves to write about Conversion Optimization.
Find him on Twitter: @NitinDeshdeep

Comments (4)

Leave a Comment

Leave a Reply to Richa Gangwani Cancel reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes : <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Contact Us / Login

Product
Resources Home