Powering Customer Experience Optimization with speed, involvement, and knowledge sharing

Maurice is from Click Value, one of our partners in the Netherlands.

Yeah, Maurice, I’m really curious to see what you brought with us, and I’m confident that we’re going to have a lot of takeaways. Before you go ahead, just one small reminder: the good thing about an online webinar, and the advantage for the people who are living with us, is that you can ask questions. So, in your GoToWebinar tool, you should see a section for questions. Please do not hesitate to ask any questions. It doesn’t happen so often that you have such an experienced person with you, and we will jump in on your question.

Maybe while Maurice is sharing his slides with us or at the end of the session. With that said, Maurice, the stage is yours. Go ahead.

Maurice:

Yes. Thank you very much, Jan. Thanks for the introduction. Yeah. Welcome all to the VWO webinar, I’m delighted to give you a presentation about CXO, and I’ll explain to you later what that actually means.

Short attention on my side, I’m from the south of the Netherlands, working in Amsterdam, but, living in the south of the Netherlands, like Jan already mentioned, more than 20 years of experience. Now I have been working for almost 1 year at Click Value. A short introduction of click value, we are a CRO-specialized agency. Part of the LevelUp Group is on other brands on digital marketing and analytics, around us. We run about 1000 A/B tests a year. We have more than 30 dedicated CXO Pros in our team working for more than 50 countries, doing some world projects for Heineken and Grand Vision.

Also in the team, there’s a lot of experience. What I want to talk about today is to give you some clear insights on how to grow your business with CXO. And then, you know, maybe your question, and Jan already mentioned this. He was delighted that I called it CXO. And what is CXO?

CXO is customer experience optimization. And why do we call it CXO instead of conversion rate optimization? I believe that CXO has more value in it. It’s not only about optimizing your conversion rates for more revenue; when you give your website a better customer experience, revenue will grow, and customers will be happier. The CXO is all about starting to know what the intention is of your visitor. So the main purpose in our vision is of CXO, and we created a matrix, which I will show you later. The idea is that optimizers should know everything about the intent of the customer. While you know what your customer is up to on your website, you can make better business decisions and create a better funnel or flow towards the end goal of your customer. We created this matrix, and the end must be that the interesting part of optimizing websites is that even if you don’t use it, as you call it, CXO or conversion rate optimization.

People are still making a lot of changes to websites. Nowadays, there are a lot of product teams, scrum teams, and agile teams. Even when there’s no CXO on board, they still make changes because they want innovation or they want to optimize their products. That’s quite interesting because when you start with CXO, this is the situation.

There is no validation or resets at all, but there are still a lot of changes. The disadvantage of this approach is that you really don’t know if your changes have an effect or impact on your revenue or on your website goals. So, what we normally do with companies is we start with validation first. More and more companies are aligned with scrum and agile teams and always have a big backlog with about 80 or 100 IDs. When we come in and suggest doing some research, they often say, “No, we have enough; we have plenty of IDs. Please help us with validating those IDs.” 

What happens then is, when we start validating via A/B testing and running experiments, they get scared off because they see that many of their IDs didn’t have the positive impact they expected. There are a lot of tests that have a really negative impact. So, that’s the reason they start thinking, “How do we solve this? We thought we had really good ideas.” But, in this example, out of 4 IDs we validated, only one was a winning experiment. The others were losers.

We don’t call them losers. We refer to them as loss prevention because before implementation, we gain insight that this wasn’t the best idea to implement. So, we go back to the drawing board and figure out how to improve that ID. Then, product teams often ask, “Okay, what should we do next?”

The next step is that we should start both the research and validation. When you do research upfront, you have better insights into the customer’s behavior, and you see that more and more tests start becoming successful. The reason is that there are still some changes left under, and there are still changes left above. This is the situation in many companies. Some departments are doing a good job on experimentation, both in research and validation, but there are still some product teams around that just do some validation with less impact. Additionally, there are still some product teams that probably don’t believe in optimization or experimentation. They rely on a good feeling, just adding changes to the website without any validation or research.

In most companies, this is the situation, and standardizing these processes becomes a challenge. Growth is a real issue for those companies because they don’t know the effect of all the innovations and changes. In the most ideal situation, CXOs continually improve and experiment, researching and validating all their changes.

And that’s a real challenge, I admit, but that should be your end goal when you implement CXO in your company. So, the story is stuck now.

Jan:

Something happened to your presentation, I think.

Maurice:

Yeah. Next slide. 

That’s happening with webinars. Excuse me for that. This message about validating and researching everything isn’t a new message. It’s about the message Jeff Bridges pronounced years ago. He said, “Our success is a function of how many experiments we do per year, per month, per week, per day.” There’s another famous person with a similar mindset. If you only validate 10% of your business changes, you can’t make any impact when 90% are still implemented based on gut feeling and opinions. I think this is the main issue with CXO at this moment. From my point of view, this message is echoed by successful companies like Netflix, Amazon, Microsoft, and others. They all emphasize that conducting a lot of testing and research leads to more business value and success. However, this isn’t the common approach for many companies.

To make your customer experience optimization program a success, you need to increase the number of changes you validate and research. This way, you gain a better understanding of customer behavior and which changes are effective. In CXO, we often talk about velocity and volume. But having a high velocity, like 50 tests a month, doesn’t matter if you have 500 changes that are unvalidated. The philosophy is really adapted to the number of changes added to your website without validation. This is an important metric for your company.

Now, there’s an important question: How do we get there? That’s nice—more validation, more research.

You were already mentioning Booking and Netflix. Many people wonder how this works for smaller companies. That’s a good question.

Back in 2015, I wrote a Dutch blog about transitioning from a MacGyver to a team. Today, I want to add more information about how to evolve from an individual specialist to a team in a CXO program where many departments collaborate and start testing. For those who didn’t read my blog years ago, here’s a short introduction to what I meant.

It often starts with one CRO specialist. In a recent survey among our newsletter readers, we were surprised to find that a large number of companies still have only one CRO specialist handling it all. While it’s commendable, being the sole CRO specialist has its drawbacks. As the lone MacGyver, you may handle CRO programs, UX Design, and maybe some easy “What You See Is What You Get” editing, like validation with simple A/B or copy tests.

However, there are disadvantages to this approach. The one MacGyver often forgets the basics, getting so busy with executing at least two A/B tests a month that they neglect data integrity, Google Analytics issues, tagging issues, performance issues, and more. Development is not always on their side, and many basics are forgotten in this solo approach. There are things a MacGyver should focus on when starting a CXO program on their own to become a successful CXO specialist.

It all starts with a logical, solid process. You really should start archiving your learnings, add people to the program so that they put their ideas on a backlog, and you can start aligning and do some stakeholder management on that. Also important, you have to be very precise on analytics and statistics. When you make rookie mistakes and make mistakes on the minimal detectable effect, you make statistical mistakes with the validation of your experiments. There will be one way that people will notice this, and then they may dismiss your information because there were mistakes in it. So, there’s a lot of responsibility for just one person, and it’s hard to let the program grow in this way.

What you really need to do is, okay, it’s good that one person has started the program. But we really should go to the next phase because you really need a team for real growth. And that’s why my story in 2015 ended for more value and velocity. You meet more people. And this is how we often start.

And the nice thing is when you are a CXO specialist and you see people becoming very enthusiastic, that one designer or that one developer, please add them to your team, even if they are not really in your team or in your department. Create an environment around CXO, and people who want to help, add them to your team. But normally, this is the team you need a lot to create some bigger growth than compared to the one CXO specialist because you need specialists. I always say, where you have people who call themselves CXO specialists, the only one in the company, they are not really specialists. They are generalists because they combine all the specialties into one person. But for a good CXO team, you really need specialists. You need a researcher. You need a UX designer. You need a developer who can help you develop more difficult tests. You need an analyst because it’s necessary to validate and evaluate your tests. And you need a program manager to make the program bigger within the company.

There’s only one problem with this approach. The more visible you get with this approach, the more stakeholders come in. Because you have success, maybe you have a few very good tests with a yearly value of 1,000,000 euros. Then stakeholders come in. The QA officer comes in, questioning if it’s all safe. Maybe the copywriter from the communication and branding department comes in because they saw you run a test that didn’t align with branding standards. The scrum teams start asking why you tested on that product detail page without informing them. A lot of people start asking questions or want to be part of the program because you are a successful program. But that also means you get a kind of, sounds heavy, enemies, as you need more alignment with stakeholders. 

I don’t know if you’re familiar with the story of the A-Team in the eighties. They were suspected of robbing a bank, and that’s why Colonel Decker, the military police, pursued them. He’s always chasing the A-Team. When you run a CXO program, there will also be Colonel Deckers in your way. There might be no development capacity, or development isn’t allowed to help you. There could be heavy brand restrictions. Based on research, we’ve found that changing some brands’ styles leads to more success, but we’re not allowed. You are permitted only to test usability because testing on new product features is not allowed; that’s the area of the product teams.

As you have more success and gain more insights, hitting the product features and services becomes inevitable. You, as a company, start adding business value. However, as a kind of standalone team, people might face problems in terms of their position in the company or whether they are seen as working. So, while this is nice for a quick start, we don’t need to see those people as enemies, but we have to be confronted with everybody, including the Colonel Deckers of our company, and we have to make another approach.

Working with specialists is very good; it’s beneficial for a quick start. However, our goal, at Click Value, is always to involve more people inside the CXO program. We often provide insights, like the A-Team, to help start up a CXO program. But our aim is always to have more people involved in the program for better outcomes. We are always looking for real cooperation with all the teams that matter in the CXO program. So, if we truly want to make more impact and influence products and services on our websites, we need to think bigger.

The thing we have to do is adapt our organization for CXO to become a success. We need to change our approach, go back to the Colonel Decker program, and become friends with all the formal organizational units responsible for products and services. This is a new chapter I started writing in 2015, and now in 2022, that’s a new chapter, a power of multiplication.

When we want to do this, we enter another area—change management. The MacGyver and the A-Team could run tests on their own. Now, we want existing departments to run tests with us. That’s a different approach. The first thing we need to do is learn about the people we want to influence. The most important thing is to understand our brain because, as typical human beings, we have certain tendencies caused by our brain’s functioning.

One simple example is that humans, generally, don’t like radical changes. When CXO specialists suggest doing things in a completely different way, people get scared because they feel they were already doing it wrong. So when integrating CXO into existing departments, it’s crucial not to push too hard. Many people don’t understand data, and CXO is essentially a data-driven process. We deal with statistics, accounting, runtime, minimal detectable effects, and various numbers, including test outcomes based on segments. While some may find it interesting, many may not feel comfortable admitting that they don’t fully understand it.

So, the reporting of your insights should sometimes be simpler than a complex report with a lot of data and insights. People might say yes, but they really don’t understand. It’s a valuable insight to consider when adapting departments on your site that haven’t had experience with A/B testing and data. In addition to dealing with various data, we also encounter psychological aspects like biases, paradigms, and cognitive dissonance. People tend to believe in patterns, and it’s challenging to change those beliefs.

Sometimes, there’s a lot of misunderstanding when introducing new insights and beliefs into new departments. It’s not that people don’t want to do it, but they have strong belief patterns, and adapting to a totally different way of working takes time. This subject could easily be the basis for another webinar, but it’s important to recognize that change management is hard.

The last thing that’s very important is to know your culture as well. Two English scientists, Quinn and Rohrbaugh, in the early eighties, already invented this matrix.

And they said, in companies that have different cultures: family culture, innovative culture, hierarchical culture, and market culture. For example, a market culture is one where there is a “show me the money” attitude. People in this culture want to earn money quickly, and money is all that counts for them. I once made a mistake when explaining to the management team of a company with a market culture that we had conducted many tests and gained numerous learnings. Despite having no winners yet, the learnings were very promising. However, the management team became angry, demanding to know where the money and winners were. They expected immediate results, which didn’t align with the nature of the CXO program. So, before you start expanding your CXO program, it’s crucial to know the kind of culture you have.

For example, in a culture with more hierarchy, you should be aware that you need to convince three management layers before starting tests. This knowledge is essential for planning and stakeholder management. Every culture has its unique aspects that you should understand before initiating changes. Keep in mind that, while we aim to validate as many changes as possible, we may face challenges when introducing CXO to different departments. I refer to this approach as the CSI approach, drawing parallels with the TV series CSI, where different teams in Miami, Las Vegas, and New York have distinct characters but a similar scientific approach to solving murders.

And it works the same for defining CXO among different scrum teams or agile teams. But the question is, how do you add value to those teams? Because they already have a way of working. Most agile scrum teams work in this way: they conduct tiny research, sometimes do sprint poker or voting in the teams, and they are very democratic about which team will work on a particular sprint. They choose a team, quickly design a solution, reach an agreement, finalize the requirements, and move to implementation. Developers implement it on the websites, and there’s cheering because many product teams are held accountable by the number of interfaces or improvements they add in the biweekly sprint on their website.

When you come in as a CXO specialist, many product owners may think that you’ll start changing their process. What helps significantly is to communicate that you’re not changing anything. Instead, you’re bringing something new. You come to bring research and validation. The existing process with design and implementation remains intact, but now you bring research beforehand to help them better understand what to design. After the UX design, you validate the idea, and then they can proceed with implementation. This approach resonates well with many product owners.

This will work, but they still have a lot of questions and challenges when starting to work this way. Typically, teams want things to be fast and quick, especially with biweekly sprints. There’s an expectation that something must be delivered by the end of the sprint. However, when you introduce research and validation to a product team, particularly when dealing with high traffic and conversions, experiments may take around 4 weeks to run. For instance, 1 week for upfront research and 1 week for design. This means it takes about 5 weeks before they can begin the implementation phase.

This is a matter of planning, and it becomes a real challenge. When you understand what they want to achieve in, let’s say, week 12, you need to start the research part around week 7, begin tests in week 8, and then, by week 12, have the results to decide whether to proceed with implementation. Planning becomes crucial, and there are different solutions for this, such as creating a separate lane for designing research and validation or integrating research and validation directly into the sprints, even though it may delay the development phase. It’s also essential for product teams to learn how to conduct proper research. When they see the value of research firsthand, they begin to realize its significant contribution to their way of working.

So it’s not only the CXO specialist coming in and telling them what to do; the scrum team is doing their own thing, and the CXO team is working behind that. Now, you really have to be convinced that researchers should be a part of scrum teams. This is crucial. I’ve always emphasized that many product teams focus on completing projects. They celebrate with cake and coffee after delivering sprints, and they’re happy. But when you ask, “Why do you celebrate this?” The response is often, “Because it’s in our OKRs or KPIs.”

The game would change completely if the KPI were to celebrate when the implementation is a real success. This is something you can’t tackle within the product teams alone; it needs to be addressed at the management level. How are the product teams being held accountable? Are they accountable for delivering an innovation, or are they accountable for delivering a successful innovation? That’s a significant difference.

Another crucial point is that when you want cooperation, you need to be wrapped up in it. For instance, based on the old A-team approach, you align with communication and marketing, involving all departments. To convince product teams to work in a certain way, CXO needs to be part of product scrum, agile teams, or even development. You need to be there, one of the team members, and they’ll accept your message more easily.

The ultimate goal in the coming years might be to automate all testing because, when you agree on testing every change, development can create an automated tool that validates everything with each change.

In this way, some parts of Booking.com are already working. They have so many changes that they can’t handle them manually, necessitating automation for smaller parts. However, they genuinely want to know if a change has any impact, whether negative or positive. This aspect is crucial to remember, especially when dealing with, for instance, 12 product scrum teams.

Having a clear dashboard and archiving learnings becomes essential because, with each team running an average of 5 tests a month, you end up with 60 tests a month. How do you gain an overview of all that’s happening? The key is to have good reports and start reporting on the various outcomes. When multiple teams start working on CXO, such as 12 different teams, you can almost validate all your changes, which is a positive development.

However, the story is not complete without addressing the management of these diverse teams. When you have 12 teams, even if they follow the same agile scrum methodology, the challenge is ensuring that they all work within the same CXO methodology. How do you achieve that alignment across the teams?

Because there’s still some work to do here, you’ll need to tell the CXO story company-wide. There are some new guys coming in who are guiding and teaching these newcomers. You have management over 12 teams, and you also need management for the overall program because there are many winners but also numerous challenges. You have to manage your costs and budgets. As I mentioned earlier, there are learnings that you want to document. And, of course, you have to think about innovations.

When you start manually with all those tests, there’s a smart guy who suggests, “Okay, we really should automate the simple copy test.” The question arises: who’s responsible for this innovation? Is it one of the 12 scrum teams, or do we need an overall program? The risk, if you don’t take care of the overall program, is that CXO programs may start independently, each teaching and innovating on their own, resulting in a lot of overhead and expenditure. In my opinion, the CXO team should run their own experiments and have their own show, but the focus should be on experiments.

For this, we should create a team of specialists, and maybe these are the friends in the story. We create characters where we need a marketer to tell the story of experimentation, a teacher to guide and educate, an operational specialist for metric program management, and someone who archives all learnings and shares them. What is discovered in, let’s say, team 3 can be easily reused in team 6. I’m sure of that. So, you need individuals for overall innovation. This forms what is called the Center of Excellence, a crucial element when you have multiple companies running meters. This approach is not new, especially if you are in a company that has adopted scrum or agile methodologies.

You also have central Scrum and Agile Masters who help the Scrum teams improve their processes. This is based on the Scrum Agile methodology; a center of excellence assists separate company departments in becoming better at optimizing and validating websites. That’s a logical step, but it’s often forgotten in many companies. So, in this example with 12 different Scrum teams, sharing and communication between the teams are crucial. They share not only within the teams but also with people who are not directly involved to create awareness. Share when a test starts, when a test ends, and when you have results. Keep informing everyone, and if possible, start sharing information at an earlier stage. When the central CXO team is involved, it’s also a good track to use because sharing information helps people understand what you’re doing and why. This alignment and understanding garner more support from others. This is the route to impact: start on your own, create a central team, distribute responsibilities among different teams, and establish a support network with friends who assist multiple teams.

One more thing. That’s a very important one, right? Get a grip on the cost of A/B testing. When you’re in a situation where 12 teams run (I gave the example of 5 tests each a month), you have 60 tests. When they’re all running very complex, big development-wise tests, then the cost of the 60 experiments will, on average, be very high, costing you at least €5000 each. These are high costs, and then you need a lot of winners and a lot of learnings to gain them back. With a lot of traffic and conversions, that’s easily done. However, when you’re going this way, you need to think about automation, simple copy tooling. This is one area where the VWO tool is capable of providing more democratization to testing so more people can easily add tests and IDs. It’s already proven by many CXO colleagues that it’s not always the biggest test that creates the biggest winner; sometimes even a small ID can result in a significant win. Also, retesting is crucial. When you do 60 tests a month, usually, half of the tests can easily be retested in other periods or in another scrum team. So, sharing knowledge, ideas, and results is important. In summary, remember that your main goal is to validate and research as many changes as possible. That should be your primary objective. When you do that, you need specialists; you can’t do it on your own. It’s logical to start with one team, but if you want to validate all your changes, you need more people on board.

Then you need to start adding CXO to your product or service teams—teams where business decisions are made and where products are changed or developed. These teams really need validation and research to become better digital products. Lastly, as you start growing in CXO, you really need a Center of Excellence for the unity and quality of your testing approach. So, this was my short story, and I hope you liked it.

Jan:

I like it a lot. Actually, I liked it a lot. Having worked— you know, what I liked most is having worked for 5 years in an agency for conversion rate optimization and customer experience optimization. There are a lot of agencies out there who still try to keep, let’s say, the experimentation work very much on the agency side, right?

Okay. And not, let’s say, hand over the necessary skills and get people on this trip towards becoming friends, right? I think that’s really describing very well the journey that we would highly recommend as well. And I think it’s that— that’s also why, to a certain extent, is why, 10 years back or 12 years back when the VWO was created, they had in mind to create something with the front end that everybody could use this democratic process and so on. It won’t be immediately you need specialists, especially at the beginning, and then you can move it on into different product areas.

And it’s not a thing that you do in 3 months. It’s longer— it is definitely a longer journey. And there is definitely— it definitely should become a priority. We analyze the status quo in different countries, and there is a big difference between, well, first of all, the United States, and all the other countries.

So they are way more ahead on this road. And then in Europe it is in the— so what would you guess? What is the leading country of Europe where you have the highest maturity on this road that you laid out? What would be your best guess for us?

Maurice:

I would guess the Netherlands or the UK, one of the others.

Jan:

It’s the Netherlands, actually. I wasn’t aware before that, but there, it’s clearly the most advanced country where you find the most advanced companies on this experimentation roadmap, closely followed by the UK. Then the German-speaking region and the Nordics come along. But there’s a big difference; there’s still a lot of things to do, which is okay.

As we said initially, I wanted to ask our audience, ‘Hey, guys. You’ve got some interesting questions for us.’ And, as I also said, it’s close to lunch, so everybody’s a little bit tired and looking forward to dropping off and having a bite, something like that.

So I threw in my suggestion that I would personally invite the person who comes up with an interesting question, which will be answered in this webinar. And guess what? It worked. So, my hypothesis is confirmed.

I have good news for Stephan. He actually came up with a question that I really liked a lot. If you start as a conversion rate optimization specialist and eventually move on to a team with different roads, what is or what should be the role of the original conversion rate optimization specialist?

Maurice:

Ah, that’s a good question. Yeah. And it’s really the most easy answer—it depends. It’s worth it…

Jan:

A pizza. No. I think it’s worth a pizza. I would say so.

Maurice:

Continuing from your input:

Yeah. But it depends because, where, yeah, where is your house, where you are when you’re starting as a CXO specialist, you have some background in analytics, maybe design, maybe you are a great project manager. And I would say, where is your heart? So personally, I also ran some projects in the past, totally on myself, but for me, I’m more like a project manager. So I really like when I’m in the team, I really like to lead the project. But there are others who are both product managers, designers, or really good analysts. And, you see in a lot of roles that the data analyst is often around who starts using experiments to get more insights. So it really depends on how you would like to fulfill your role.

It’s not worse that, in the team, you are the analyst and someone else is taking the program management role. So, I would say, do what your heart desires, do what you think is good for yourself. And, yeah, for looking at Click Value, we are always very happy with web analysts, for instance, because there’s always a shortage of them. So, yeah, every role in a team is important. You can’t do without UX design. You can’t do without program management. You need development. 

Jan:

Yeah. I have seen a lot of programs in the last few years where this particular person was exclusively or almost exclusively working on conversion rate optimization where they, these people grow in their roles, and they become project managers, and they become people managers as well. So they become more visible in the organization teaming up with other people. And, that is why very often it is from becoming an or moving from being an expert, to becoming a manager of a program. So you’re actually then managing resources and relationships and so on and so forth.

Maurice:

You also see a lot of double roles in a lot of teams. For instance, the one is the program manager and the researcher or the program manager and the analyst. In most teams who are starting, you see that design and development already existed somewhere so that you can add them easily to your project. But research and validation, that’s the biggest challenge to find those people within our company.

Jan:

Okay, Maurice, that was really insightful. As you said, just drop me an email to Maurice at clickvalue.nl. I think it is.

Maurice:

Yeah. Yeah.

Jan:

There’s a typo. There’s a missing dot in nl. Also, I am available; I have the shortest email here, jan@vwo.com. So, let us know. Thanks for being with us.

Thanks for staying with us. I’m really amazed. I’m looking at our list, and nobody dropped off. Mauricio did a great job. Hope we can meet you back with your next story there and looking forward to more joint action in this field.

Thank you very much. Have a wonderful day, everybody, and see you soon.