How To Scale Your Optimization Program With Automation

But today, we will talk about something very different that entirely focuses more on increasing the activity part of the optimization program. A lot of us understand APIs are something that send data from one platform to another. And typically, with respect to VWO, we’ve seen a lot of our customers use APIs to send testing data to an analytics platform or to their BI tools, but we often underutilize the value of API in terms of productivity. Just to give a quick example, most companies out there have optimization teams, wherein team members have different responsibilities, for instance, some will be responsible for doing research, and then you’ll have developers, designers. 

Teams often use agile methodologies like running sprints, doing stand ups, using IT project management tools, but, you know, there is a better way to scale optimization programs by automating most of these operational works and increasing productivity. So to help you understand the same with an actual case study, we have Gino and Thom from Merkel. So I will let Gino and Thom talk more about the journey of using VWO APIs at Merkel. Please feel free to post your questions, which we’ll take at the end of the webinar. Over to you, Gina, and Thom.

Thom:

Hey, everyone. My name is Thom. As you can see, I’m a data tech and optimization consultant at Merkel. I’ve started at Merkel, and have been working at Merkel since August 2018. Ever since I’ve started there, I’ve helped multiple organizations with their optimization programs.

I’ve built experiments, done research, analyzed experiments, and helped clients with the strategy of their optimization programs. That’s it for short.

Gino:

Yes. And hi, everyone. I’m Gino. I’ve been working with Merkel for a little over 2 years now. I do a lot of optimization stuff, similar to what Thom described, but I’m also very much invested in marketing technology.

And as you will see in this webinar, the area where these two meet is, and if our project has originated, We’ll get into that later.

Thom:

Yeah. Let’s see. So, do we need to talk about the questions and the way people can ask their questions? Hello?

Rahul:

Hey, everybody. Please feel free to ask questions during the webinar. You can also post your questions on Twitter with Harsh Title SpeedW.

Thom:

So, yeah, we’re from Merkle, a global marketing agency with offices around the world. In Europe, we have offices in Great Britain, the Netherlands, Spain, and Germany. While we have many employees, our main office is in Amsterdam. There, we merged with our former partners, Frokley, Optima, and YourSocial, into Merkle Netherlands. We are still part of the digital marketing agency in the Netherlands, where we support multiple brands with digital analytics and optimization programs. Our vision is to help clients make informed decisions using data and technology, moving away from decisions based on opinion to those backed by evidence. Gino and I are part of the data, tech, and optimization team at Merkle Netherlands, where we engage in a broad range of activities.

We collect, analyze, and orchestrate data using marketing technology. Today, our focus is on the optimization aspect. Every consultant at our agency is skilled in various tasks, and that’s where the idea for the rover, or the message bots, originated. So, that’s a bit of background.

Our conversion optimization team in the Netherlands currently has seven members, and we love what we do. Today, we will discuss four specific topics. First, we’ll look at stakeholder communication. In a large optimization program, you have to deal with many stakeholders, so effective communication is crucial.

Second, we’ll discuss creating actionable alerts. We’ll cover the solution we developed to manage stakeholder communication and how we use the VWO API to facilitate this. Third, we’ll explore automated quality assessment testing, explaining how we automate quality assessments during experiments for companies.

Lastly, we’ll talk about monitoring experiments. When running experiments, it’s essential to monitor their progress. We have devised a method for this, which we will explain later. For now, let’s focus on stakeholder communication. In a large optimization program, managing numerous tests and stakeholders requires good collaboration.

Everyone needs to know which tests are running, which pages are affected, upcoming tests, research conducted, and research results. Therefore, you need a solid strategy for keeping everyone informed.

Basically, there are two teams: the optimization team and the organizational stakeholders. The optimization team includes developers, optimization specialists, UX leads, and analysts. This team needs to be updated on all actions in the optimization program. Additionally, stakeholders need to be informed. For example, a customer service lead needs to know about ongoing experiments to communicate effectively with customer service in case clients or website visitors encounter issues. While this doesn’t happen often, it helps with the acceptance of optimization within a company.

Moreover, a CXO (Chief Experience Officer) may not be directly involved in testing but needs to be updated about ongoing activities and take necessary actions. With these diverse roles and information needs, it’s essential to have a robust plan for keeping everyone informed and satisfied.

One of the frameworks that you could use for that is the responsibility assignment matrix. This is a matrix which we use to define what kind of role everyone has and what kind of responsibilities belong to that role. So, if you look at the CXO, the Chief Experience Officer, he is responsible for all the optimization, testing, and design—basically everything related to customer experience. He doesn’t need to be the one who does the daily operational activities. He’s not the one who builds experiments or analyzes experiments.

But is this someone that needs to be informed, for example? So, there are four different types of definitions that you can define. They’re responsible, which means someone is actually doing the work. That might be, for example, a developer that develops experiments. We assign them the development of an experiment.

Second, you have to be accountable. That might be someone who makes the final decision and has the ultimate ownership. That might fit the CXO role. Third, you have a consultant. There’s always someone who has good advice or needs to be consulted before you put an experiment live. This is an important role that must be taken into consideration.

Fourth, you have informed me. This is someone who needs to be updated about certain actions within an optimization program but doesn’t need to be consulted or involved in activities directly related to the optimization program. It’s just someone who needs to know what’s happening and nothing more. By assigning these four roles to each person within your team and stakeholders, communication becomes a lot easier.

While doing that, it’s really important to make sure that your notifications or communications are actionable. This ensures there won’t be too many notifications or irrelevant information. People will only get the right information. You should make them actionable so people can take action right away without having to take extra steps.

While running an optimization program and communicating with stakeholders, there are different layers of communication, which vary in scope. You have long-term communication, which might be a yearly strategy meeting to decide on the strategy for the coming year and assess maturity and areas where you’re behind. This is a long-term meeting done in person, and it happens every year.

Then, you have the weekly or quarterly progress checks. This is more mid-term and happens more often. It’s more operational, less strategic, and involves defining the activities for the next few weeks or months, and checking how everyone is doing on their tasks. It’s a mix of strategy and operations. In the short term, you have different activities that are being done in relation to the optimization program.

And this is where a lot of time is spent on these activities. While we were optimizing for larger companies, we noticed that many tasks could be automated instead of having to do them manually each time. If you can automate activities like monitoring experiments, QA experiments, or checking how many experiments were successful, life gets a bit easier. You’ll spend less time on operational tasks and more on strategy and quality experiments. If you’ve defined each person’s responsibilities, you can make it clear who needs to know what and who needs to be updated about certain actions.

If you look at the slide on the right, there’s the optimization team. Many people are actually working on activities. The UX lead might be the head of the optimization program and is accountable, while others are responsible for conducting tests. On the left side, you have people who should be informed or consulted because they are further from the team but still need updates about certain activities, such as when an experiment starts or performs poorly. 

You see a difference in how each person within the team or stakeholders needs to be updated. After you’ve done this, you know what kind of communication is needed and who needs to be updated. Then, you can start thinking about how to make this easier instead of typing updates to each person. For example, if you have to notify many people about going live with a test, it becomes cumbersome. For one client, we conducted about 80 or 90 experiments in a year. We realized we could save a lot of time by automating daily communications.

We looked at what kind of notifications or communications should be sent to which roles. For example, if an experiment (like number 250) starts, many people might need to know, such as a product manager who shouldn’t confuse an experiment with a bug. If an experiment is live but no traffic is coming in due to a targeting mistake, only the relevant people who can address it need to be informed, along with the person accountable for the entire optimization program.

This kind of planning helps define communication within an optimization program and helps us create our solution for handling lots of communication, tests, and tasks under time pressure. Now, I’ll hand it over to Gino, who knows everything about the solution we developed.

Gino:

Yes, thank you, Thom. As Thom described, we thought of a messaging system that would really help us take our CRO program to the next level. So the question for me was, how do we bring this to life? How can we make this something that actually works? The thing we came up with was Rover. Rover is a virtual assistant, a chatbot on Slack, able to mention specific people and broadcast messages.

This was the starting point. VWO has an excellent REST API available, providing a lot of information about campaigns and other relevant account details. Slack also has an extensive API available for messaging and more. That’s where we started. The use cases Thom described, like messaging people when a test has been published, were something we could achieve by periodically checking the VWO API for campaign statuses. We recorded these statuses locally, and whenever a status changed, we knew a test had been published and could send out a message. Then it was a matter of connecting the Slack API, choosing the message recipients and channels, and setting that up.

But with Rover, I didn’t want to create just another boring notification system. We had an internal chatbot that sent notifications about all sorts of things, like birthdays and verification tokens, and that boy had a bit of personality. So, we decided to give Rover a playful character. Rover is a dog that always barks for your attention whenever he sends you a notification because you need to take action. This makes it more fun than just regular system messages or emails.

We also wanted to make it interactive. The schematic on the right side of the slide shows the collection of services connected to Rover. It started with the VWO API and Slack, but we also connected Google Tag Manager for additional tracking, Google Analytics for test information, Jira for project management details, and IBM Watson for smart capabilities, which I’ll get to later.

The first system we built around Rover was its notification capability. Whenever something happened regarding a test, Rover would broadcast it to the relevant person. In the screenshot, you can see that if I’m the person disabling the test, Rover would send me a personal message. This notification system worked so well because it kept everyone relevant in the loop as we grew.

We soon expanded Rover. One such expansion was adding a quiz component. We used information from Jira to create tests, add background, and detail the manipulations applied. Rover pulled this information from Jira and input it into an interactive Slack message, allowing multiple people within the organization to compete. People could vote on which variation they thought would win. Interestingly, those not involved in the optimization program often predicted outcomes just as well, if not better, than seasoned optimization specialists. This highlighted the importance of having an optimization program. It’s a great way to engage people involved in testing and those who support the process.

And it’s just a great way of showing what experiments are live or are about to go live and what is happening here and just show people who are not really involved with optimization, how it works. And really draws them in as well. And so this is also a good starter, for people to then start submitting ideas, for example.

Thom:

Yeah. And oh, sorry. If I can add something, quizzes are great ways of, as Gino said, making people enthusiastic about optimization programs. But when you make them enthusiastic, they will also start to generate their own ideas and give them to the optimization team because they will soon understand that decisions shouldn’t be made on the basis of opinion, but based on data.

If they see that with quizzes, not everyone gets it right all the time, and there’s a lot of randomness in the number of times people get it right, they’ll start to support the optimization culture more inside an organization. So it’s something very small, but it’s very powerful when you apply this inside an organization. So it was really great that Gino could pull this off and that we had this.

Gino:

Yeah. It’s just really fun that this brings the concept of optimization to life for people who only have a vague understanding or no understanding of it. And yeah. So this is a really, really fun project to work on as well. Another area that we investigated was, through the quiz element, there’s already some interactivity with Rover.

So you have the buttons that you can press, and that’s how you make your choice. But because Rover is a messaging bot, he’s sending you very direct messages about what is going on. Sometimes this can be quite actionable. So if Rover notifies you that a test isn’t running well, it could be very handy if you have it on your phone, to just stop the test straight away on your phone and then find the time to investigate. But that originally wasn’t possible.

So what we did, we linked IBM’s Watson to that, and what Watson allows you to do is set up dialogue flows. We can interpret simple requests from the user and link those to the specific API calls for VWO. In this case, because we are working with the test IDs, it’s very easy to link them to the relevant ones. An option through the API as well is to receive more information or link that to Jira, but in this case, it sufficed to have Rover notify you, and you can direct him to do something right away. There’s also some intelligence here that can use the VWO API to check if a test is already off, for example. So in this case, he would notify you if you tried to disable the wrong test. If you made a typo, it would be easy to fix.

I just wanted to say that this kind of slows us, already into the, need to have some quality assurance there because, in the example on the previous slide, it’s very easy to make a typo and then do something that you don’t necessarily want to happen. And when you are part of a large optimization organization, you run a large number of tests. It’s almost unavoidable that things could go wrong. And because of this, you just really want to have strong QA in place. And the thing with that is that it can get cumbersome, running QA on a high number of tests.

So we wanted to automate as much of this as possible. So anything you would like to add to Thom?

Thom:

Yeah. Yeah, QA-ing is something that is, especially in the beginning of organizations, further optimization programs, is not done properly. But even if you do it properly, yes, still mistakes are made and it costs so much time and money if an experiment has been put live. And after a few weeks, you look back and you see that this experiment hasn’t been running as you would have expected. Even though we experienced it, of course, it’s just a bummer if you notice that. So, what if we could apply the 4 ICE principles, which means that every experiment that goes live has been checked by someone else as well who was not part of making that experiment go into testing neutral.

What if you could make that a bit easier? And that’s, I think, where the next idea came from.

Gino:

Yeah. So moving on. We have some rules in place that whenever you want to publish a test, it has to adhere to certain conditions. I don’t think those are too difficult to follow, but it’s easy to make mistakes. And it’s also easy to miss those mistakes.

What we noticed in the VWO API is that we could get a lot of information that we would always check for anyway. So we could just write some automated checks on that. What we have here, as an example, is that you could query Rover to examine a specific test before publishing it. We would just check for these things. It doesn’t mean that we wouldn’t check for other things manually or anything, but these are very costly mistakes if not executed properly.

For example, with naming conventions, we have systems in place that work only if the naming convention is correct. So if the wrong test ID were in the name, that could cause problems. It’s just things like that that are really nice to have, just an extra layer of security that before publishing, you know, okay, at least these things are alright. Something that we’ve also built into Rover was set after publishing a test, so that you just send a reminder to the person who’s responsible for that test to investigate in GA and VWO if everything is going as expected. Just to be on the safe side here.

It’s easy to miss something, and in this case, Rover is a little helper to make sure that the experiment’s running well. We took this a bit further as well. So in addition to notifying people of just having to check if there is any problem with a test, we also thought of the idea, what if Rover could do this check himself? What if we’re not completely reliant on our own actions anymore, especially with the volume of the test pricing? So we included the Google Analytics API in the mix.

What it does here, it checks for the events that we have pushed when a VWO test is active. So in this case, Rover detected that for test 233 after a day or so, there is not enough information available, which makes it likely that the test isn’t triggering correctly. That is something that’s really useful to have, as Thom said. If you only noticed this after multiple days, or even multiple weeks, you’ve wasted a lot of time, possibly a lot of money. It’s an issue.

We could also expand Rover to have him notify you of experiments where one of the variants appears to be performing rather badly. He can make some comparisons to the control and other variants. That way, it’s just easier to intervene quickly. The only thing here is that what we don’t want Rover to do is give automated updates on how different variants are performing because you do need a lot of data for this, and we don’t want to make him send out notifications prematurely because it doesn’t really work that way. Thom, how do you see this?

Thom:

Yeah. Checking continuously on experiments, how they’re doing is something that we see very often as clients, but something that you shouldn’t do. Before you start an experiment, you have to calculate the number of visitors that need to be in your own experiments before you can call the experiments or the results of experiments significant. And if you start to continually peek every day, then the chance is very high that you’re gonna make the wrong decision based on that experiment because you’ve been checking the results of the tests.

Statistically, you really shouldn’t be doing that, which is also why we haven’t even though it’s really easy. We didn’t make the feature that would check the results every day and calculate the conversion lift and push that to slack because yeah, it’s not, it shouldn’t be part of an optimization culture.

Gino:

No. And it’s something that people are very interested in learning about, but we Yeah. It’s just not a good thing to do, and we don’t feel that a notifier like a broker should be used as a platform for that. What I hope that we’ve kind of been able to show during this webinar is just that Rover is really a supporting element, it doesn’t really replace anyone within the team, but it does take a special role within your team.

So what we noticed with Rover was that he really became one of us, like, one of the team. It’s very fun to see something that in reality, not more than a bunch of lines of code. Yeah, just to really be part of an organization and really be part of something, something that you do. And it’s also very interesting to think that if there are expansions in the VWO API, for example, or if there are other surfaces where a power could be derived from, it’s very easy to plug those in. And especially now as Rover is also conversational that you are able to interact with him.

It’s just a very cool project to have worked on. And I think many organizations would benefit from having something like Rover on board or even just utilizing the API beyond reporting purposes. Anything that kind of wraps it up for me, Thom? Anything you’d like to add?

Thom:

No. I think Yeah. When you were talking about the fun part, it’s really important, when you’re creating and sending all these notifications to everyone that you shouldn’t make a system that’s gonna be ignored right away because it’s just because it doesn’t have personality. It sounds really strange, but, in the time that we used Rover, we really looked at each notification because we also built in some randomness, what would say, which means that we were more engaged with the notifications, and we were really up to date on what the other colleagues in the optimization team were doing. Which meant that, yeah, we definitely had a more effective optimization program.

Gino:

Yeah. That’s definitely true because the risk of sending a lot of notifications is just everything getting ignored straight away or getting boring. And we hope that through adding this persona to Rover so not making him just a generic notifier, we’re really connecting with people. And I guess what we’ve seen is that that actually happened. So that’s also something to keep in mind.

Yep. And I think that’s mostly it on our side. Very curious to hear what you guys want to ask us. If there’s anything extra that you’d like to know?

Rahul:

Thank you, Gino, and Thom, for this wonderful session. And, you know, I found a few key use cases very, very interesting, especially where, you know, you guys talked about the quizzes and also the QA part. Because, you know, working with so many customers over the years, you know, I’ve realized that a lot of people are usually stuck at the QA stage as well. So we, on a daily basis, get a lot of queries around, you know, tests not being able to run or where they’ve done something wrong during the setup. And I feel, you know, all these automations where you’re able to QA your tests right there before starting can really help you be more productive.

And, at the same time, you know, in terms of collaboration, I felt that the quiz part was really interesting, and I’ve never really seen anybody do something similar to sort of, you know, keep the entire team engaged and collaborate around great tests. So I think it was a wonderful session. Thanks a lot. And, we’ll quickly take up questions. So we have a lot of questions. Let me take the first question.

So this is an interesting question from Paul. He says, how did you get the inspiration to build Rover? So, Thom, you want to take that up and then probably, Gino, you can also add here.

Thom:

Well, I think because Gino built Rover, and he had the idea. I think he wants to answer this one, and I’ll add to that.

Gino:

I don’t think I’ll take this one. So, I think already at a younger age, I was kind of interested in these chatbots, but when MSN was still a thing and I was in high school, I was working on little bots that you could play games with and such. And so I think the inspiration to integrate virtual assistants or chatbots into my work has always been there. And I think just in the case of Rover, it also happened rather naturally because we were doing a lot of these notifications manually.

And we also had some struggles with that, and we noticed that when we were the ones doing the notifications, for example, somebody else was publishing tests or making modifications, it was difficult to keep everybody up to date. So then to me, it seems kind of natural to automate that. Also, having worked on several other automations within the digital marketing scope, it seems natural to combine it too, and that’s how the idea for Rover was born, really.

Rahul:

Perfect. Thank you, Gino, for that answer. I think I have a similar question from Brian, and he wants to understand that, you know, usually people go for an IT project management tool to manage their optimization program. Then how does, you know, your sort of you also mentioned that you are using Jira. So how does Jira and the rover work together?

Gino:

I think they complement each other here. So, Jira is really a project management platform. And, I think for the purposes that we have here, it could be a bit much. And, I think the notifications within Jira also work a little differently. So, I just think that Rover makes it very, very accessible to be in the loop on what is happening around tests and also to automate that part and to add that personality to it, as we described earlier, which you don’t really have with Jira. If you’re anything like me, you probably get dozens of Jira notifications in the day. And it’s easy to overlook them and some are a bit time-sensitive, like when tests go live. And that’s not something that you want to miss. So we use them in conjunction with each other. So Jira has the place where we do all the management for the tests and where we have it moved to the column board. And then when it comes to everything that’s surrounding the publication of the test and then the initial results for that, that’s where Rover steps in. So I think where you want to keep that is really a matter of preference. But in this case, the way that we have Rover set up and the way I see it, it’s more complementary for accessing project management.

Rahul:

Alright. Thank you for the answer. We have another question from Alice. She wants to understand that a robot must have taken a lot of development time as well. So how did you convince the stakeholders to invest in building something like a robot?

Thom:

Yeah, I’ll answer it, I guess. Yeah, it definitely cost a bit of development time, but we were really convinced about the power of such a tool and the structure of Rover. So it uses Slack and VWO and uses Google Analytics and Jira. It’s a stack of tools that we see at all of our clients as well. So we knew that we weren’t developing it for just one client. We were making it for multiple clients and that convinced us to do it. Do you know what you want to add to that?

Gino:

Yeah. And I think on the client side, it was very easy to get started on Rover because it was just apparent that the QA was difficult to control without automations. And, I think it was just easier. It’s easier to convince somebody of the use of such an elaborate project if it is actually going to immediately solve a problem.

And so our problem was that it’s difficult to keep everybody in the loop. We have some QAing that gets more difficult and that we would also have to spend increasing amounts of time on. And so in a way, that has really helped us just to prove the value of our offer as well. And then as Thom said, being able to roll them out for multiple clients, to keep the cost low for just one, that works out.

Thom:

And, of course, we didn’t build this in one day. We built it as we went. We didn’t Yeah. So we didn’t have to spend a lot of time upfront, but we spread it out over a couple of months to do this. And we also already have some experience doing this because we also built a lot of dashboards and do a lot of data orchestration.

And, over there, we also connect with various APIs and do error handling and stuff. So we’re kind of familiar with this kind of activity. So that made it a Gino’s set very natural to think of a solution like this.

Rahul:

Oh, makes sense. So I’ll quickly take up another question from Jesse. The question is, the robot includes a connection to IBM Watson for NLP. Can you please explain how the NLP is playing a role here? Gino you know you wanna take that off? Understand the role of NLP, in Rover?

Gino:

Yes. So, what we really wanted to do to make him interactive was, we wanted people to be able to have some communication with him. And so the communication with him isn’t that complex, but we didn’t want to just hard code a set list of commands to him and then have people speak those exactly because otherwise, he wouldn’t be able to pick up on that. And so Around that time, it was just something that was brought to my attention. I thought it would be interesting to just experiment with that.

And, I think I had used Watson on a previous project once before, but that was more experimental. And so here is what I really like about using Watson is that it makes it easy to set up the dialogues that you can have with the bot that you are creating, and also that it’s easy to extract kinds of information from what is being said. And so what we really see is being used in Rover, for example, the test IDs. So it makes it really easy to work with those and then take actions based on that. And she can do it in a relatively safe way. It was just a good fit.

Rahul:

Thank you so much, Gino, for the answer. I think we’ve already crossed the time. So one final question, what is the future for Rover? So this question is addressed to both of you. So I think if both of you could give your last words around future plans for Rover.

Thom:

While developing Rover, we made a roadmap document where we listed all the different features that we wanted to add. So there’s a list of very interesting features that we want to discover if it’s possible to pull off. Also, when the VWO API has some more options in the future, then we’ll also look at that to integrate. So, yeah, I think Rover won’t be the same in a few years’ time.

Gino:

No, exactly. I think Rover is only going to get smarter. So I think that would be very interesting to see how we can fit more AI into that, for example. I think its basic functionality will remain the same, but it has the potential to evolve into something much bigger, and because it’s built fairly modular, we could also add other platforms into there that could be interesting to us that we are not really considering now.

So I think that’s really where Rover will go, just to grow its importance within the team and the sky’s the limit, really.

Rahul:

Makes sense. So I think, you know, we’ve already crossed the time. And thank you again, Gino and Thom, for enlightening all of us with such wonderful use cases, which I’m sure most people have never heard before. And to everyone who’s joined the webinar, thanks a lot for attending this webinar.

We will take a breath of the questions line over an email, and we’ll ensure we’ll answer all your questions, and feel free to shoot any questions to VWO on Twitter, and we will ensure that we answer all these questions. Thanks again, everybody, for joining, and we’ll end the webinar now. Thank you so much. Bye bye.

Gino:

Thanks to all for hosting us and bye everyone.

Thom:

Yeah. Thanks for hosting us.

Rahul:

Thank you.