VWO Logo
VWO Platform

Conversion Rate Optimization Solutions for Revenue Growth

VWO’s end-to-end CRO platform helps brands understand visitor behavior, make observations, build a testing pipeline, execute the tests for statistical results, and engage through new-age channels.

Know more
Complies with:
VWO GDPR Ready Badge
VWO CCPA Ready Badge
VWO G2Crowd Leader Spring Badge
Trustradius badge for VWO

We tweet useful stuff daily

Related content:

Meet Rover: The Chatbot that Scaled CRO through Automation

8 Min Read

Continuous experimentation and testing increases conversions at high growth and data-driven organizations. Scaling a conversion rate optimization (CRO) program requires not only building a long-term testing roadmap but also building a culture of experimentation involving multiple teams and stakeholders across the company. As the scope of CRO projects increases, so does the difficulty of management and the tediousness of the manual efforts required. To overcome these pain points, digital marketing agency Oogst – a Merkle company collaborated with HEMA, a leading Dutch retail chain and created a chatbot by bringing together the power of the VWO REST API and Slack, to scale CRO efforts at HEMA.

This blog post shall delve deeper into why and how we leveraged the power of VWO API and Slack to scale our CRO program and help in building a culture of experimentation.

The challenges of scaling CRO

For Oogst[1] and HEMA[2], CRO involved constant juggling of interests between its CRO specialists, analysts, clients, website developers and other relevant stakeholders. Each decision and action affects many people, which is why the process needs to go as smoothly as possible.

For example, whenever a test is activated or stopped, many people need to be informed at the very moment it happens without leaving anyone out. When you scale CRO program at your organization, informing everybody manually becomes a hassle, especially with new tests being turned on and off frequently. This process needed to be automated to bring efficiency and agility to the CRO initiatives.

The Birth of Rover

We discovered that the VWO REST API makes it easy to monitor tests for changes in status, and that we can use this information to make announcements easily. As a result of these discoveries, the idea for a chatbot was born. By linking the VWO API, Slack webhooks and a local database through Python, we were able to notify the right people about test status changes the moment they occurred. We named him Rover: he would be a dog/bot hybrid that barks at us when it needs our attention. Rover also sends notifications to relevant parties about when to check the preliminary results of a test. A huge burden was lifted from our shoulders, leaving us with more time to think about the next set of strategic experiments.

Growing the bot

As these simple adjustments to our progress proved to be quite helpful to us and HEMA, we began exploring the VWO API to see what else we could do. We came up with a number of areas where the bot could help us with:

Quality control

We run a high number of tests and it is of utmost importance to us and HEMA that everything goes according to plan at all times. We have set up quality control checks internally to limit the risks. There are processes we follow before publishing a test to make sure it’s safe to go live and we double check right after publication in case anything was missed.

For example, some of the things we check for on the VWO side include if the traffic has been set to 100%, if GTM integration has been enabled, and whether the campaign’s name follows our naming conventions. Luckily, the VWO API made it possible for us to automate the checking of settings like this, so that we can be more certain a test is ready to be published, while taking us much less effort.

screenshot of the conversation with the Slack Rover bot

Monitoring live A/B tests

After publishing any test, we monitor events in Google Analytics to make sure we didn’t miss anything and the test is, in fact, running correctly. Although it allows us to maintain the level of quality we desire, this process is also very time consuming, tedious and prone to human error, much like the announcements.

To deal with this, we added the Google Analytics API into the mix to get Rover to check for the amount of VWO events for a particular test and notify us about its findings. The absence of VWO events likely meant the test wasn’t running (correctly), which is something we always had to check manually before.

screenshot of the reply from the Rover Slack Bot

Conversational A/B test management

However, at this point, Rover would only transmit messages, he could not process received messages. If we wanted a user to intuitively tell Rover to check a test, we’d have to include an element of interactivity.

That’s why we hooked up our bot to artificial intelligence using IBM Watson[3], which allows it to naturally process language. With this integration in place, a user is able to ask “Rover, is VWO test 244 ready to go live?” and it will perform the checks. Then, the user can simply publish the test by saying “Rover, publish 244”.

Rover Slack bot sharing the update for the A/B test

Right now, Rover has truly become part of the team, both of Oogst and HEMA. His contribution to the speed and quality of our process is well recognized. We included even more functionalities such as the ability to pause the manual activation tags we built in Google Tag Manager to trigger VWO tests, thereby linking VWO and GTM together.

But we’re not finished yet!

Scoring points: Building a CRO culture through gamification

Although the power and effectiveness of CRO and A/B testing are well established, building support for it throughout the organization still remains a difficulty.

To combat this, we found yet another role for Rover: quizmaster!

First, we pull basic information about the test and its variants from VWO. We then use this data to extract more detailed information about the experiment from our project management board. This includes background information, hypotheses, descriptions of the control and variants as well as screenshots.

From this knowledge, we dynamically generate polls on Slack where users vote on the landing page variants they think will outperform the rest. These users, generally members of the client organization, then battle each other in a CRO tournament where it is determined who knows most about the website visitor.

screenshot of the CRO tournament conducted by the Rover Bot in the Slack channel

The introduction of a gamification element has made CRO more exciting to those not generally involved in it, it has made our efforts more visible throughout the organization and also highlights the successes we have. Furthermore, it builds team spirit and generates more ideas for future testing. All because of our data-driven quizmaster: Rover!

To sum up:

Here’s a quick summary of the benefits we have already achieved:

·         Notify stakeholders of starting/stopping of tests;

·         Notify stakeholders to check preliminary results;

·         Check Google Analytics for presence of VWO events;

·         Perform quality control checks;

·         Pause Manual Activation tags in GTM;

·         Start/stop tests through chat;

·        Perform the role of quizmaster, allowing stakeholders to vote on which variant they expect to outperform the others.

More to come!

Future plans

We’re in touch with VWO to expand the abilities of the REST API even further to make Rover more powerful. This collaboration is a very exciting one to us. The API has already proven to be invaluable in our current operations which are now running more smoothly than ever before.

This bot is the collaborative effort of Gino Renardus, Martijn Heerema and Thom van Engelenburg (consultants at Oogst, a Merkle company) and Floor Hickmann and Raun Sips (UX at HEMA).

About Oogst

Oogst, A Merkle Company, is the leading digital marketing, analytics and optimization agency in the Netherlands. Based in Amsterdam, its team of over 70 experts provide leading brands with digital marketing services aimed at utilizing customer data in the best possible way to achieve the highest returns. The company has partnered with VWO since 2012 to test and optimize some of  the most popular websites of the country for their clients. Their extensive knowledge of, and experience with the VWO platform has led to these operations continually growing in size. Moreover, this has also led to increased stakeholder involvement and more intensive test management. In order to better deal with this increased scope, the Oogst team was recently joined by a new member: a CRO Chatbot that operates through the VWO Application Programming Interface (API). This chatbot is able to assist in many of the activities surrounding the CRO process.
Do you want to meet Rover or get to know more? Get in touch with Oogst’s Data, Tech and Optimization team! Reach out via gino@oogstonline.nl.

About HEMA

Since 1926, HEMA has made the everyday life of its customers easier and more fun through products that positively stand out: due to their quality, design and price. HEMA offers over 30,000 of its own products and services, has over 750 stores in nine countries and 19,000 employees. As consumers move more towards online, HEMA recognizes the importance of developing a digital strategy as progressive as the brand is. HEMA’s webshop is widely recognized to be one of the best of The Netherlands, with a top position in the Twinkle 100 and by frequently receiving awards such as Best Department Store Webshop. HEMA and Oogst collaborate on CRO to ensure its Dutch and international webshops remain among the best.

Gino Renardus
Gino Renardus Gino Renardus is a Digital Marketing Consultant at Oogst, A Merkle Company, where he is part of the Data, Tech and Optimization team.
More from VWO on Conversion Rate Optimization
What Does a CRO Program Consist of? | An Interview with Avast’s Michal Parizek

What Does a CRO Program Consist of? | An Interview with Avast’s Michal Parizek

The following is an interview with Michal Parizek[1], Senior eCommerce & Optimization Specialist at Avast[2] (a…

Read More
Nitin Deshdeep

Nitin Deshdeep

9 Min Read
Thinking Like a Bayesian

Thinking Like a Bayesian

I recently finished reading a book on the history of the Bayes’ theorem (appropriately called…

Read More
Paras Chopra

Paras Chopra

3 Min Read
3 Most Important eCommerce Metrics- AOV, CR, RPV

3 Most Important eCommerce Metrics- AOV, CR, RPV

It’s important to understand that metrics simply show symptoms, and different symptoms become visible through…

Read More
Shanaz Khan

Shanaz Khan

8 Min Read

Scale your A/B testing and experimentation with VWO.

Start Free Trial Request Demo
Shanaz Khan from VWO

Hi, I am Shanaz from the VWO Research Desk.

Join our community of 10,000+ Marketing, Product & UX Folks today & never miss the latest from the world of experience optimization.

A value for this field is required.

Thank you!

Check your inbox for the confirmation mail