ISM eCompany Helped Staatsloterij Increase CTR by 8.25% using VWO
VWO and ISM eCompany
Headquartered in the Netherlands, ISM eCompany is a well-known full-service digital commerce agency. It uses future-proofing and distinctive data-driven digital marketing techniques to help its clients achieve their digital and optimization goals. ISM eCompany has been using VWO Experience Optimization services since 2011 for all their A/B testing and conversion optimization needs.
In a recent conversation with Loef Kleinmeijer, Team Lead Conversion & CRO Consultant at ISM eCompany, he shared his experience using VWO and some of his learnings from the recent experiments he and his team ran for their clients.
Elaborated here is a multi-goal experiment ISM eCompany ran for their client Staatsloterij, a Dutch-based state lottery ticket company. Staatsloterij provides people with an opportunity to win huge prize money through a lottery draw system.
Staatsloterij’s primary objective was to provide comprehensive information about its XL option and get more people to select it during the checkout process to increase the lottery ticket size.
The XL option allows an individual to play twice with the same ticket number, win prizes from the monthly prize package, and also win prizes from the XL prize package.
The execs at Staatsloterij saw that XL transactions were consistently low on both desktop and mobile. Despite the presence of a tooltip, not many people selected the option. They approached Loef and his team to understand the problem and seek assistance on increasing XL selections and lottery ticket sizes.
Loef and his team conducted an in-depth analysis of the XL option using VWO’s capabilities. They used heatmaps, sessions, and other qualitative research tools to understand why visitors did not select the XL option.
They discovered that visitors often encountered the XL option for the first time on the shopping cart page. Many even clicked on the XL toggle and tooltip out of curiosity to learn more about it. However, the tooltip hardly provided any valuable information about the XL option because of which most visitors switched off the XL option and proceeded to check out.
Loef and his team hypothesized that by adding more information about the XL option in the tooltip and explaining why visitors should choose the option, they should be able to increase the XL option selection and the lottery sale amount.
They used VWO Testing to run a Split URL test. They created a new version of the checkout page for both desktop and mobile and added two elements on the checkout page.
- An arrow with text to increase the focus on the XL toggle option.
- More information about the XL option in the tooltip and why visitors should select the option.
Here are the control and variation for desktop and mobile, respectively.
The team ran the test for 17 days on both the platforms, with traffic split equally between the control and variation. Here’s what the team analyzed:
- The number of transactions increased by 1.04%.
- Visitors clicked less on the tooltip, possibly because the arrow focused much attention on the XL toggle option than the definition of XL.
- Visitors switched on the XL toggle more than switching it off (+ 8.16%).
- The arrow with text focused on the extra chances of winning, which served as a motivator.
- The XL explanation had a negative impact on the number of transactions (-1.10%.)
- Visitors clicked on the tooltip less often.
- Most visitors switched on the XL toggle more than switching it off (+ 8.35%).
- The number of transactions compared to the desktop was less, possibly because of the arrow and text position.
After observing an uplift on the desktop, the team at Tele2 deployed the change and it’s currently live on the website. Meanwhile, an uplift wasn’t observed on their mobile platform, and hence, the team is currently contemplating to run more experiments with different designs to achieve their objective of more XL selections.
ISM eCompany runs every experiment with one primary end goal: getting an uplift in revenue through successful A/B test implementation. They plan and strategize in a manner that every experiment run provides enough learnings for them to grow and achieve their goals.
VWO is well suited as an experimentation tool for many of our mid- and large-sized clients. It offers an easy setup and implementation, both for CRO specialists/analysts and developers, sufficient functionality and segmentation options for a relatively economical price