Meet Rover: The Chatbot That Scaled CRO Through Automation

This is a guest post by Gino Renardus, a Digital Marketing Consultant at Merkle, where he is part of the Data, Tech and Optimization team.

Experimentation and testing has the power to increase conversions for any business. But for high-growth companies, scaling a conversion rate optimization (CRO) program requires not only building a long-term testing roadmap but also a culture of experimentation across multiple teams and stakeholders. 

Download Free: Conversion Rate Optimization Guide

As the scope of a CRO program increases, so does the difficulty of management and the manual effort required. To overcome these pain points, my agency Merkle, collaborated with HEMA, a leading Dutch retail chain, to create a chatbot by bringing together the power of the VWO REST API and Slack.

The challenges of scaling CRO

For my team at Merkle and our client HEMA, CRO involves constant juggling of interests between CRO specialists, analysts, clients, website developers, and other relevant stakeholders. Each decision and action affects many people, who have to be consulted before and informed after any changes. This can become cumbersome for small businesses but particularly affects larger CRO teams, especially with new tests being turned on and off frequently. This process needed to be automated to bring efficiency and agility to the CRO initiatives.

The birth of Rover

We discovered that the VWO REST API makes it easy to monitor tests for changes in status, so we sought a way to use this information to make announcements easily. As a result, the idea for a chatbot was born. By linking the VWO API, Slack webhooks and a local database through Python, the chatbot was able to notify the right people about test status changes the moment they occurred. We named the bot Rover: he would be a dog/bot hybrid that barks when it needs attention. Rover could also send notifications to relevant parties about when to check the preliminary results of a test. A huge burden was lifted from our shoulders, leaving us with more time to think about the next set of strategic experiments. 

Growing the bot

As these simple adjustments proved to be quite helpful to us and HEMA, we began exploring the VWO API to see what else it could be used to do. We came up with a number of areas where the bot could help automate routine tasks:

Quality control

When running a high number of tests, it is of utmost importance to us and HEMA that everything goes according to plan at all times. We have set up quality control checks internally to limit the risks. There are processes we follow before publishing a test to make sure it’s safe to go live, including double checking right after publication in case anything was missed. 

For example, some of the things we checked for on the VWO side include if the traffic has been set to 100%, if GTM integration has been enabled, and whether the campaign’s name follows our naming conventions. Fortunately, the VWO API makes it possible to automate the checking of these kinds of settings, so that we can effortlessly and quickly be more certain a test is ready to be published. 

Monitoring live A/B tests

After publishing any A/B test, we monitor events in Google Analytics to ensure nothing was missed and that the test is, in fact, running correctly. Although it allows us to maintain the desired level of quality, this process is also time-consuming, tedious, and prone to human error, much like the announcements. 

To deal with this, we added the Google Analytics API into the mix, allowing Rover to check the number of VWO events for a particular test and notify us of its findings. The absence of VWO events likely meant the test wasn’t running correctly, which is something we always had to check manually before.

Conversational A/B test management

At this point, Rover was already hugely valuable to the CRO team, but it was limited to only transmitting messages. Rover couldn’t process any received messages. If we wanted a user to intuitively tell Rover to check a test, we’d have to include an element of interactivity. That’s why we hooked up our bot to artificial intelligence using IBM Watson, allowing it to process natural language. With this integration in place, a user is able to ask “Rover, is VWO test 244 ready to go live?” and it will perform the checks. Then, the user can simply publish the test by saying “Rover, publish 244”.

Rover has truly become part of the CRO team, both of Merkle and HEMA. His contribution to the speed and quality of our conversion rate optimization process is well recognized. We included even more functionalities such as the ability to pause the manual activation tags we built in Google Tag Manager to trigger VWO tests, thereby linking VWO and GTM together. 

But we’re not finished yet!

Scoring points: building a CRO culture through gamification

Although the effectiveness of CRO and A/B testing are well established, building support for it throughout the organization still remains a difficulty. 

To combat this, we found yet another role for Rover: quizmaster! 

First, we pull basic information about the test and its variants from VWO. We then use this data to extract more detailed information about the experiment from our project management board. This includes background information, hypotheses, descriptions of the control and variants as well as screenshots. 

From this knowledge, we dynamically generate polls on Slack where users vote on the landing page variants they think will outperform the rest. These users, generally members of the client organization, then battle each other in a CRO tournament where it is determined who knows most about the website visitor.

The gamification element has made CRO more exciting to those not generally involved in it, making our efforts more visible throughout the organization and also highlighting the successes we have. Furthermore, it builds team spirit and generates more ideas for future testing. All because of our data-driven quizmaster: Rover! 

Download Free: Conversion Rate Optimization Guide

To sum up: 

Here’s a quick summary of the benefits we have already achieved:

· Notify stakeholders of starting/stopping of tests;

· Notify stakeholders to check preliminary results;

· Check Google Analytics for presence of VWO events;

· Perform quality control checks;

· Pause Manual Activation tags in GTM;

· Start/stop tests through chat;

· Perform the role of quizmaster, allowing stakeholders to vote on which variant they expect to outperform the others.

Future plans

We’re working with VWO to expand the abilities of the REST API even further to make Rover more powerful. This collaboration is a very exciting one to us. The API has already proven to be invaluable in our current operations which are now running more smoothly than ever before.

About Merkle

Merkle is a leading digital marketing, analytics and optimization agency in the Netherlands. Based in Amsterdam, its team of experts provide leading brands with digital marketing services aimed at utilizing customer data in the best possible way to achieve the highest returns. The company has partnered with VWO since 2012 to test and optimize some of the most popular websites of the country for their clients. 

About HEMA

Since 1926, HEMA has made the everyday life of its customers easier and more fun through products that positively stand out: due to their quality, design and price. HEMA offers over 30,000 of its own products and services, has over 750 stores in nine countries and 19,000 employees. As consumers move more towards online, HEMA recognizes the importance of developing a digital strategy as progressive as the brand is. HEMA’s webshop is widely recognized to be one of the best of The Netherlands, with a top position in the Twinkle 100 and by frequently receiving awards such as Best Department Store Webshop.