How SEAT does business experimentation: Lessons for marketers

Both Wouter and Anouk have spent years learning experiments at SEAT and companies like it. In this session, they are going to share the nuggets from their experience along with some real examples of tests that they’ve done for SEAT. With that, Wouter and Anouk, the stage is yours.

Wouter Sportel:

Thank you, and welcome everyone, to this webinar. Before we start, we would like to give a quick introduction of ourselves. So, my name is Wouter Sportel. I’m Head of CRO conversion rate optimization at Accenture in the Netherlands. And therefore, I’m focused on doing a lot of experiments, doing research, in target audiences, setting up experiments, learning from them, and also giving advice to our clients. And therefore, we are setting up multiple optimization processes at different clients all over the world.

And today, we are going to share some interesting takeaways on business experimentation where we focused on the last couple of years, say, up in the Netherlands. But before we start, I would also give Anouk to introduce yourself as well.

Anouk Donkers: 

Yeah. Thank you very much at first for all joining this webinar. It’s a pleasure. My name’s Anouk. I’m working at sales in the Netherlands, and I’m responsible for online marketing, which means brands and performance advertising, websites, and journey optimization. And on a daily basis, we are working with experimentation to make better business decisions.

So that’s what this webinar is about. Of course, I’m very excited to share all lessons together with Wouter today, and we will start with sharing the agenda with you guys.

Wouter: 

Yes. So first of all, we’re gonna, well, give you the definition or our definition on what business experimentation actually is. After that, we come to the second part where we talk about why you should invest in business experimentation? What are the main benefits of it? Then we go straight into some practical, business experimentation examples where we also share different kinds of experiments which we actually did or are now doing at the, say, out in the Netherlands. 

And, after that, we’re gonna focus on how to start with business experiments yourself. So you, as an organization, where should you, how should you be able to start with business experimentation in your organization to also make impact and get relevant insights, regarding this topic. And last but not least, we finish with a short summary. And obviously, there is some room for questions also.

So if you have any questions during our talk, please add them in a chat. And after we had other subjects, we will come back to the questions in the Q & A in the last part of this webinar. 

So once again, welcome everyone, and we can start with the first part of the webinar. And that’s our definition for business experimentation. So what is it actually?

So business experimentation is all about experimenting to make better business decisions. And as the word already is saying, business decisions, so it’s relevant for all kinds of organizations. 

So, this kind of experimenting is different compared to AB testing, since you are focusing on, for example, new strategies you would like to set up, or new propositions you would like to validate, or even products or services that are not well available right now, or which maybe don’t even exist right now. And you would like to validate those kinds of things before actually implementing them.

Since, obviously, it could be a really risky effort when you think about the new proposition, it will immediately publish it to the markets. However, you don’t actually know how your target audience will react to this new proposition. Therefore, business experimentation is a really interesting way of validating before actually implementing. Testing new strategies, propositions, or even products and services is one thing about business experimentation. But you obviously can also optimize in your current environments.

So, testing new innovations on your website or within your app—adding new features to your website or app—before actually fully developing it and pushing it live. Since I think you all know, and if you’re working at a big organization, you will probably have a development department which, in most cases, has immense big backlogs. So, for those kinds of situations, it’s also really interesting to first test the pilot and then see what the impact actually is before fully developing it in the team, since it will become part of the backlog, and maybe it can also take a lot of time before it’s actually implemented. And you won’t even know what the effect actually will be when it’s boosted live. So, therefore, business experimentation is a really interesting way of testing new features and innovations before bringing them to the backlog of developments.

And last but not least, it’s all evidence-based. So, when you do an experiment or a business experiment, the insights you will get are actually evidence for you—as a marketer or as an organization—to make better business decisions. So every experiment you do, it will give you insights. You will learn from it, and you will be able to give advice or make a conclusion out of it. And also be able to, well, make a better decision before fully developing it or before implementing a new proposition or strategy.

So, that basically is the three parts of business experimentation. And why are we doing this? Well, it’s a really important thing to obviously experiment, since if you are not experimenting, you’re just guessing what will work or what won’t. Also, a lot of companies in our digital marketing world are using experimentation as a product development framework. For example, when people are focusing on product development, it’s really interesting to do that in an experimental way. Since obviously, when you push live new products or bring a new product to the market, and you didn’t test it or you didn’t do any research, market research, beforehand, it will be just a guess if the products will, well, be irrelevant for your target audience.

And with business experimentation, you can actually validate this. For example, Amazon is doing this a lot. Actually, Jeff Bezos said, “Our success at Amazon is a function of how many experiments we do per year, per month, per week, and even per day.” So, it really shows how many experiments they are actually running to improve their products and their websites every day. To give you a practical example, if you’re living in the Netherlands, you will notice an example for sure. Lay’s, a big brand, conducts those kinds of business experiments themselves.

So last year, I think this campaign ran and they brought some new flavors of their product to the markets in different supermarkets in the Netherlands. And people were able to vote on the flavor that they liked the most. The flavor which got the most votes will actually be in development for the rest of the year, for example, or maybe even longer. It’s just a really nice example of a business-related experiment where Lay’s first wanted to validate what kind of new flavors they should bring to the market. They let people vote on their favorite flavor to know what flavors they should take into production for a longer period of time.

I think this is a really interesting one and also gives Lay’s the evidence and the learnings about which flavor will work best for them and make the most money, obviously, for them as an organization. So, that is business experimentation, and our definition of it.

But let’s now focus on why you should invest in business experimentation?

Anouk: 

Yeah, now you know what business experimentation is, I would like to tell you more about why. Why is business experimentation interesting for your company, and why does it help you make better decisions? Well, it’s kind of a no-brainer that optimizing your existing products and creating new innovations are essential to stimulate sustained success. The world is changing, your audience’s needs are changing, and you need to adapt.

And experimentation can help you with that. It can help you validate those new ideas, those new optimizations or innovations, and check if they are actually effective like you have in mind. I like to look at business experimentation as a continuous loop where you can start with a new idea, for example, think about the new proposition, and test experiments with a new idea but with a new proposition to see, okay, how does my target audience respond to that? What can I learn from the data that the experiments result from? And based on that learning, you can optimize the ideas.

So, you are learning along the way, and you can build a base on data. To sum up, if you skip the experimentation, it’s just a hit or miss. You are building the ideas once, and you see, okay, does it work on us? It could, of course, cost you a lot of money. But if you invest in using experimentation, it can still be a hit or miss, but you at least get the evidence and say, okay, this works. I will go further and build the idea on, or I just bin it and don’t go further with it because I know it doesn’t work.

So, the three benefits of business experimentation are, first of all, it’s lean & mean. You can start small and don’t need a large investment. Also, you can test and learn. Like I already mentioned, you can learn along the way based on how your target audience reacts to your idea. And last but not least, you can build it successfully.

You can build it based on the data you collect with the experiments instead of just gut feeling, where most ideas start. So, that’s actually why you need business experimentation. And now, I think you might be thinking, okay, this sounds interesting, but can you show us some examples? Of course, we can show you some examples.

The first one is an example of validating a new proposition; in this case, a test drive at home.

In a normal situation, people book a test drive online on our websites and then go to a dealership to experience the car of their interest. But during COVID times, it was a challenge to actually experience a car since, obviously, dealerships were partly or fully closed. The advice for staying at home, so people don’t tend to go to the dealership and experience a car, was just not for now. And we saw that test drives dropped; test drive requests dropped during that period of time. 

So it’s like, okay, how can we help that consumer who is now looking for a car and wants to experience the driving experience of their favorite car? So we thought, okay, what if we bring the car to them? And we come with a test-at-home proposition. So that’s exactly what we did.

And the goal of this experiment was, okay, let’s get insights and see if people are willing to book a test drive at home. So what we did is we added a checkbox to a test drive tool where people can check the boxes and say, okay, I would like a test drive at home, and the dealer will come to you. We also added some relevant referrals on relevant pages. For example, when you complete a configuration, you land on a 7-week page where you can see everything about the car you configured.

And this is a place where we normally refer to a test drive since you configured your car. This is the car, how you like it, and how you want to experience it in real life. If you want to drive, if you want to feel how the car drives. So that’s what we do in normal situations; in this place, we are referred to as the test drive tool.

And what we did during the experiment is just a simple piece of copy where we added the new proposition. So the test drive at home. We told the people, okay, it’s now possible to take a test drive at home, which will be interesting. Just check the checkbox in the test drive planner, and you’re good to go. And this way, we were able to validate if people were actually interested in a test drive at home. Nowadays, we have the test drive at home proposition rolled out for all dealerships

And even when COVID is still there, regulations are, of course, different. We see that now, people are still booking a test drive at home. So it’s actually successful. Then we have another example of validating a new proposition. In this case, online sales. In the normal situation, people are looking online, orienting for a new car.

And when they have actual interests, they are going to the dealership for sales information, experience the car, like I just mentioned, and when they are ready to buy, they are actually buying or leasing the car at the dealership. The deals are closed at the dealership. So we thought, okay. But what if we make it possible to lease a car directly online? You don’t have to go to the dealership.

You can arrange your lease contract fully online. So what we did is we built an online shop environment with just a few pages, just a few steps that you need to take to arrange your lease contracts. Just lean & mean. No big investments needed. And we did it with 1 model, 1 streamlined car, and an exclusively online available model.

So it was small. We focused really on getting insights and not building the whole online show for our models and just seeing what happens. So we focused on experimenting. And, the goal was, of course, to see, okay, are people actually willing to purchase a car lease contract online? And here we did the same as with the test drive example.

We built an online shop. And next to that, we’re on different relevant pages on our website to see, okay, can we trigger people to go to the new proposition? In this case, online sales. For example, what we did is we added a new content item on the stock locator. The stock locator is a place on my website where you can find cars that are now on stock, cars that you can drive within a few days.

And since the online proposition also had the USP that you can arrange the lease contact fully online and you can drive it within 10 days, it was really fast. It was a relevant place to push the new proposition and to see if people are interested in stock. Are they also interested in arranging their lease contracts fully online and then driving fast? So, this was a way to validate if people are interested in online sales by rolling out a lean & mean shop and lean & mean referrals. This way, we were able to validate that, and now we are currently building further on the online sales proposition. We have now two models available for online sales. So, it’s getting bigger and bigger, and we’re learning more and more by just experimenting.

Then, another example, a different one this time. Within this example, we validated a new feature, a filter functionality. When people have interest in a specific model, they go to a model page, and on this model page, they find, of course, more information about this model. We have several trim lines, several variants of the car, and people are going to compare which model and which model trim line fits their needs. In a normal situation, we show an overview of the car, trim lines, and features of trim where users can compare. In the experiments, we included a new filter functionality, which makes users able to choose. Okay, I want navigation, I want Apple CarPlay, and the field of functionality shows only the trim lines that also fit their needs.

So, they have [indistinct] and navigation. The idea was that this way we make it easier for users to choose the trim line of their needs. Yeah, the goal here is, of course, to learn if people are actually interested. Are people engaging with this filter functionality? And does it help to result in a higher conversion rate?

And here you see the overview. Once again, I know I’ll also edit them into the slides, but to give you a good idea, on the left side, you see the normal overview of trim lines. On the right side, you see the same overview but with the filter functionality added. And, in this case, we started small. We built it for one model.

We built the MVP within VWO, so we didn’t use developments. That’s something we are doing now. We know this works, so we are using development to roll this out for all models.

Number 4, we’re validating future components; in this case, the login portal. It’s currently not possible to log in within our website for existing clients, and it could be very interesting. We believe that we can improve the user experiments if we add a login portal on our websites. This way, we are able to let current sales clients log in and see all the information they want – think about all the car information about your car, the next service appointment, the service history, your lease contract, etcetera.

Also, it could be a nice way to identify users in a better way. Looking forward to the cookie-less era, of course, it’s more and more important to identify users on our websites. So what we did is maybe not what you think because we didn’t actually build a login portal. We experimented by user engagement. So what we did is we added new content items about this log-in portal to relevant pages.

For example, the service and after-sales section on our sites currently mostly attract users that are driving a sale, looking for more information. So it’s really simple, and let me show you guys how it looks. On the left side, you see a normal service page, just with more information about service and after-sales.

On the right side, you see that we added a content item where we trigger with the message, ‘Are you already a serial driver?’ showing a used piece of a login portal and actually showing a CTA to go to the portal. Of course, when people are clicking on ‘go to the portal,’ they don’t actually open up a portal because we didn’t build one. But when people are clicking, we know that they are interested in this kind of portal. So we learn, okay, there’s interest; we can go further and build with the idea.

And, of course, we didn’t want a negative experience. So we showed a message when people were clicking on the CTA and told them about the experiments and, of course, thanked them for their interest and told them it will be built in the future. So in my opinion, this is one of the best examples of business experimentation because you can start really small. You don’t need any investments; just add a content item on your website and see, okay, how does your target audience react? Do they respond well? And then build further along the way based on the data you collect from the experiments. 

And then we have another example, and Wouter will tell you more about that.

Wouter: 

Yeah. Since there were some examples already about new propositions, about new features on websites, and even this new login portal, these are three kinds of different forms of business experimentation. And the last one, I also think is really interesting since we were also able to, for the login portal, also conduct a survey, for example, to get insights into what people expected when they clicked on the item to go to a login portal. Why did they choose to do it in my environment, which wasn’t available at that time? This also gave us much more insights, allowing us to focus and decide whether to prioritize this new login environment on our development backlog. Those are some nice results we got out of that specific example.

But we have another two examples for you.

And these are from our colleagues at Škoda, also in the Netherlands. The fifth example I’m showing you is also a new form of product experimentation or business experimentation, which is about validating new products. The situation for Škoda in the Netherlands is that there is no charging station available at the current Škoda webshop. It’s good to know that many car brands in the Netherlands also have a web shop available. Within that specific web shop, you can find accessories, new tires, navigation updates, and all kinds of different products.

As many of you will know, I think that more and more cars will be EV cars or electric vehicles. Therefore, charging stations are also upcoming in the market, especially in the Netherlands, where more electric vehicles are becoming prevalent. So, there will also be more and more charging stations. But the fact is that currently on the Škoda website and even on the sales website, there is no charging station available within the web shop. But Škoda itself, as a brand, is asking themselves, ‘Is it valuable for us? Is our target audience actually in need of a charging station on the Škoda webshop in this case?’ The original situation is clear. There is no charging station available at the current Škoda webshop.

What we did within this experiment is that we actually added a fake product to the Skoda webshop. I think you’re already guessing what kind of fake product this is – obviously, the charging station itself. 

So, our goal in this experiment is to gather evidence of how many users would be interested in this product before actually adding it as a real product to our web shop product range. You can imagine that if Škoda as a brand decides to sell charging stations, they’ll have to set up different contracts with partners, make stock for the charging stations, and update web pages based on different charging stations, etcetera, etcetera. Adding a new product to the web shop, especially a product like a charging station, is a significant amount of work compared to an accessory like, for example, a cap of Skoda.

You can imagine that’s a really different kind of product. What we actually did was the following. On the left side of the screen, you see the control situation – the original situation of the Škoda webshop if you go through it right now. We are in a section for electrical driving. 

It’s all in Dutch screenshots – sorry for that. That’s in this specific section, obviously, since a charging station would be really relevant within this section. In the original situation, you see different kinds of products that are relevant to electrical driving. But we added these fake products about the charging station for your electrical car. We also took a price, which would be pretty close to the actual situation, but the fact is that we obviously didn’t offer this product to our users or clients yet.

So, what happens when you click on it? If you click on that specific product, you would go to a product page. However, that product page wouldn’t show you an option to add the product to your basket. It would mention something like, ‘Thank you for your interest in the charging station at Skoda. But unfortunately, we are not able to sell it to you right now since we are evaluating if it’s relevant for our target audience or something like that.’ And therefore, obviously again, we didn’t want to give the user a negative experience. 

We decided to give them a discount code which they could use within the same webshop for other kinds of products they would like to order. This test is currently active, and we are really interested in seeing what the results will be. This is another example of business experimentation where we actually experiment to determine if this new product is relevant for our target audience and if people are willing to order it on the Škoda website in this case. A really interesting part with low efforts, as a pilot experiment, but the learnings we will get out of the experiments will be the evidence we need to decide for Škoda if we want to actually sell charging stations on the web page. So that was the last one about product experimentation.

So, this is another example of validating a new feature as a business experiment. And this one is also about Škoda’s websites. And it’s focused on the brochure you can download about the car. So, obviously, Škoda has multiple cars available on the website. And if you would like to get a lot of information and detailed information about that specific car, you are able to download the brochure on the website of Škoda.

And within the brochure, you will find a lot of information about all the different specifications of the car, all the kind of colors they are available and all the car, kind of wheels you will be able to select and so on, and so on. A lot of information about that specific car trim, on the Škoda website. And you can download it within the brochure. So that’s the original situation. But we also wanted to do an experiment on this, since Anouk also mentioned, we currently are living in a more cookie-less era.

And therefore, it’s really hard to be able to give users on our websites a personalized experience, for example. Even more, when you do not have a login environment yet, for example, where one of the other examples was all about it. So, to give you a short explanation on this cookie-less era. Before when cookies weren’t deleted after a short time, we were able to recognize users during multiple sessions on our websites due to the fact that they had a cookie in place within their browser on our client-side test. And therefore, we were able to, well, personalize the experience since we were able to recognize the user again and again on multiple sessions because they had the same cookie.

But nowadays, those cookies are, well, really short in their lifetime since we’ve intelligent tracking prevention on all those kinds of developments in the browsers. We are not able to have that cookie anymore. And, also, we are not able anymore to recognize the user over multiple sessions on that specific cookie. So, therefore, it’s really interesting for Škoda and also for sales as an organization to be able to recognize their users on a different identifier. An email obviously could be one of them.

So, we wanted to experiment with this, not only to give the user a better experience but also to get insights into what would be the benefits for Škoda in this case. Was testing with a new feature or even the proposition good? Or make both. Is mailing a brochure through yourself? So not being able to immediately download it on the website of Škoda, with mailing it to your email address and receiving it, obviously, in your inbox.

So, the goal in this case was getting evidence, how many users would be interested in this feature to be able to make a business case for user recognition in this case.

So, how it looked on the website. These are 2 completely different screenshots, and that’s all because in the control or the original situation, you would go to a specific page on this Škoda website, for example, of the ENYAQ IV, where you can download a brochure and even other pricing list information and so on. When you click on a link like this, you will be able to click on the elements and downloads to the browser itself.

But what we did in the variation, you won’t go through the specific landing page regarding the browser downloads. You would see this little pop-up, which says mail a brochure or a pricing list to your email. Therefore, we were really interested if we will see a lot fewer downloads since you can imagine if we ask users to give some information to us before they can actually download, it could be the case that there are fewer downloads or a lot fewer downloads.

But on the other hand, can we get more information out of the user and give them a better user experience by giving it to their email, or sending it to their email inbox and what will be the value of that? So, it could be the case that where we will not get fewer downloads, but the added value of the information we get for the user is more valuable and therefore, more worth compared to the original situation. And this specific example is also currently running.

And it’s a really nice example of business experimentation, if you ask me, regarding user recognition, including this new feature, which could be developed by the development team. Also, to give you an indication, if you would like to set this up and just build this, we won’t be able to see if we get fewer downloads, or yeah, we actually would but after the implementation. And now, we first would like to test it to get those insights, learn from it, and make a better business decision.

So, these were six examples of business experimentations. And as you can imagine, these are all different examples, and we can, we get all different learnings out of it, but all those different kinds of learnings will help us as an organization to make better business decisions, and decide on what will work best.

Some of them will be implemented. Some of them won’t, and the example Anouk was giving about the test drive at home is actually implemented since that was, really, working well. But it could be the case that this one about making a brochure to yourself won’t ever be implemented, and therefore, we saved a lot of money at our development department, for example. So, I think we gave you some interesting examples. And now it’s also, I think you can also imagine, I would do this at my own organization. How can I start with business experimentation?

And that’s what this part is all about. So, we will also share our key challenges we’ve faced in the previous years when we started with business experimentation since I think we had some really interesting learnings on that. Not only on the experiments itself but also on setting up this process, and therefore, we would like to share these challenges with you.

So, the first one was about getting stakeholders on board. Obviously, for some kinds of experiments, especially when you would like to publish new products which aren’t even existing, like the charging station, it is pretty handy to get approval from your manager, for example, or all other kinds of stakeholders before you actually test this. If you’re showing a product which is not available, you can’t actually do that too much since it will give a negative experience, obviously. So that’s one reason why you should get your stakeholders on board.

Another reason would be that in some cases, you would also be able to give users an alternative. Therefore, I’m also referring again to, for example, the charging station we talked about before. On that specific product page, which only said, ‘Sorry, the product is not available, but here you have a discount code.’ It’s also good to set up those connections with the departments of e-commerce, or the e-commerce departments, to make that in place and ensure that it’s actually working fine, so people will get a discount code that will also work for some period of time. This gives them the time to actually buy a product with a specific discount code.

Another challenge is that you will need to set up a team for business experimentation, and you will also need commitments from the stakeholders or the marketing communication board, for example. If you did not get the commitment on setting up a team regarding the subject, it will be pretty hard to do it in a continuous way. 

Obviously, when testing new propositions or new strategies, you will also need input from your stakeholders since you won’t be able to get those ideas all from yourself. That’s when talking about the test drive at home proposition; there are a lot of stakeholders in that specific example since you have the car dealerships. They need to be on board since, actually, they need to do the test drives at home. So those are your stakeholders. But also, the management, say, for example, they came up with the ideas to actually do a test drive at home. So there are a lot of people you will get, or you will meet on board before starting with business experimentation, and that’s the first important part about the key challenge we faced, and we would like to give this learning to you guys as well.

The second part is setting up a relevant process. Since no process is no progress, I would say it is really important to start with defining the problems or opportunities you see regarding business experimentation. And based on that, you can gather more and more insights and also be able to prioritize the different ideas you have. Since it could be the case that when you set up a brainstorm with your team with important stakeholders, I will include him as well, you will get a lot of ideas regarding business experiments. But, obviously, you cannot do them all at the same time.

So therefore, you will need to prioritize them, and it’s really easier to prioritize them when you garner some more insights out of it. The second part of the process would be setting up your hypothesis for business experimentation based on the insights you’ve got. And make sure to include also the main KPIs within those hypotheses. So the hypothesis should include the problem or the opportunity, besides that, the solution you would like to experiment with. So what would it look like, where is it focused on?

And last but not least, the KPI, to be able to also measure the results. And, for you as a marketer to be able to see at the end of an experiment, is it actually successful or not? After that, once you set up your hypothesis, it’s time to design and actually create your solution and your experiments. For example, in VWO, and then you run the experiments, validate the results, and obviously gather the learnings and make conclusions out of this. So, the third part of starting with business experimentation is actually starting, experimenting, and learning. As Anouk mentioned, you should start lean and mean and start small.

Just start with one use case, which you think is really interesting. And I’m sure you will get a lot of output from those specific experiments regarding new propositions you wanted to test, for example, but also about getting evidence and being able to gain insight into the potential of the specific experiments you did. After your first experiments and after getting the learnings about your potential, you will also be able to make a calculation. And with making the calculation, I mean, what will be the potential increase in revenue or leads or improvements in the customer journey? Because when you are able to make calculations for your first experiments you did, you will also get insight into was it worth doing this specific test.

And I think it will, since every experiment will give you some nice learnings and lessons. Then you can also go again to your stakeholders and make the statement: ‘We should continue with business experimentation since it’s getting us more revenue or more leads or an increased user experience.’ Based on all the above, you will also get more commitments from the stakeholders to continue. And last but not least, making better business decisions with business experimentation. Obviously, you will need to be able to read and find the actual outcome within the testing data.

So, a CRO specialist can help you with that, for example. And then you are also able to report the main findings and the potential outcome. Based on that, you can define your conclusions. And based on the conclusions, you can make better business decisions, eventually with the stakeholders, obviously. By doing that, you can make an impact in your own organization with business experimentation.

So, summarized, how you can start with business experimentation: get the stakeholders on board, set up your business experimentation process, start experimenting and learning, and make better business decisions and impacts. Regarding the process, a CRO process can help you out in this one.

Since business experimentation, CRO is also a lot about experimentation, you would start with getting insights out of the business contact the digital analytics, user research, for example – as three main factors for new insights regarding a proposition or innovation you would like to test. Then you start prioritizing. Based on the prioritization part, you can decide, should we test this, or shouldn’t we, and most of the items will go to the testing backlog where you can set up your hypothesis. Based on the hypothesis, you can set up the experiments, for example, with VWO, and then you start learning and rebalancing your results to be able to make better business decisions.

So last but not least, the short summary of where we talked about within this webinar. What is business experimentation?

It is experimentation to make better business decisions. It’s about testing new strategies and propositions or even products and services, but also testing new innovations within your current platform or testing new features before fully developing them and saving a lot of money in that case. And it’s evidence-based. So, the experiments, results, and insights you get are giving you the evidence you need to make better business decisions together with the stakeholders.

When talking about the benefits, it’s lean & mean. You can start with a really small investment, a really small team, and you can learn a lot by validating the different hypotheses with your target audiences on the website. You can also ensure to build new features and propositions in a proper way by doing it in a data-driven way, instead of relying on gut feeling. It’s also really interesting to do it this way since research could be a really expensive part in this case. So, these are the three main benefits.

And last but not least, as I mentioned 2 minutes before – get the stakeholders on board, set up your business experimentation process, start experimenting and learning, and make better business decisions and impacts in your organization.

So that’s it from our side. Thank you for listening. And we really like to give you the insights we got from business experimentation. We hope we shared the ideas through different theories, but also the examples we gave you.

Thank you again. And if there are any questions, we are really willing to answer them. Are there any questions, or did someone ask a question within the chat?

Anouk: 

Not yet.

Wouter: 

So let’s see if there are any questions. You can put them in the chat, but you can also ask them if that’s possible. Let’s see. I think there are no questions within the chat. So, obviously, if you have any questions after this webinar, feel free to ask them to us.

You can always connect to us, And, if there aren’t any questions, I think we are finished. So thank you again for listening, and thank you for your attention.

Siddharth: 

Thank you so much. This was a wonderful session, and we’re grateful to all the people who joined in and learnt something from this. Some great examples out there, great learning opportunities for us also. And before we conclude, I would like to ask you one thing. Are there any books that you’re currently reading or would like to recommend to our viewers or even people at VWO? We love to learn and grow more. Regarding this topic generally, other things as well?

Wouter: 

Well, there are some, some really interesting articles regarding this subject. I know that Conversion.com is an organization in the UK, which is focusing a lot on business experimentation regarding product development. If you search on Google, I think the chance is pretty big that you will find Domino’s case, so pizza delivery, that was a really interesting case. I would say since they focused really on different kinds of products within the app. And therefore, based on the engagement of users with different pizzas within the app, they made a conclusion, alright, we will take this specific pizza into development for the rest of the year, for example. So a really interesting one about business experimentation.

Also, again, in a practical way, like the Lay’s example I gave you in the beginning of this session. So Conversion.com with the Domino’s cases was really an interesting one. And besides that, you should also take a look at Conversion Excel or CXL. They also have a lot of interesting articles regarding product experimentation or business experimentation.

Anouk: 

Yeah. I’m joining with what Wouter is saying. And I mostly read articles online. I’m an online marketer, so that’s no surprise. So, mostly reading everything about online marketing from conversion to sales and branding. I’ll just say Google and you will find a lot of interesting articles and inspirational cases.

Siddharth: 

Awesome. Thank you so much, guys. So with that, I think we can conclude today’s session. Thank you so much for being here and sharing your knowledge and thank you so much to all the attendees who joined.

VWO has tons of other great resources as well for you to learn, start, and scale up your experimentation program. In case you want to start or scale up your experimentation program, please feel free to reach out to VWO and we will help you get started. Thank you so much. Bye. Have a great day.

Anouk: 

Thank you.

Wouter:

Bye bye.