How to design winning experiments

e oldest growth hacking agencies of Europe, BAMMBOO. It was incorporated in 2016 and for the last 6 years, it has created value for more than 150 companies across Europe. At VWO, we are honored to have these 2 wonderful speakers with us.

So without further ado, let me introduce you to Michel Faas and Thierry de Vries. Over to you folks.

 

Michael:

Thank you very much, Ajit.

 

Thierry:

Thanks, Ajit. I feel so honored with such an introduction. 

 

Ajit:

For everyone, I’ll clear the air. You know who is joining. I have been doing a webinar for you, but it took us a lot of struggle to have these two on board because the schedule is always tight. So we are also glad and grateful for this session.

 

Michel:

The feeling is mutual. Thank you.

 

Thierry:

Yeah. Take your time in your day to join our webinar.

We’d love to do this. What thrives us every day, doing experiments, understanding what works, our learning mindsets, and helping ourselves and, of course, our customers. So, we’re glad to share our ideas about that. Therefore, we named this webinar, “how to design winning experiments”. Of course, there’s some little clickbait in it, but it covers the topic in general.

Let’s kick off with some agenda. We have a short introduction. In this introduction, we will do a short explanation of what rapid experimentation is in our opinion and also as negatively interactive as possible. So we have prepared some polls and we’d like to invite you in the chat to respond to your vision on growth hacking.

So far, your experience with experiments has been interactive. Feel free to shout out, share, and make it as engaging as possible. For those of you who have never been to a webinar like this and only see us on your screens, feel free to fill in the chat with any questions. Ajit is here to facilitate the discussion. If something is out of our vision, perhaps beyond the rapid exploitation, there’s a definition of what it is. We also want to delve into what a growth culture is, as mentioned at the beginning. 

We’d like to learn about the growth mindset, which is an important factor in that. We would like to explain a bit about the types of companies and coaches that are out there. What will be the perfect foundations for growth and how to extrapolate this within an organization. So, what are the conditions or the pros and cons of that?

Then, we’ll show you step by step our ideal experiment process and, of course, some concrete examples. Stay with us for the whole journey. This is the part I’m excited about. Regardless of the reason why we co-created this event, we will also show you how important tooling is in this and how it can take you to the next level.

And, therefore, we give a demonstration of the tool of failure itself. We’re glad to show you that as well and put it into practice. So, let’s take it away, Michel. Right?

 

Michel:

Yeah. Let’s go.

 

Ajit:

Yeah. And before I hand it over to you guys, I have a small announcement to make. Folks, if you have any questions at any point in the webinar, feel free to post that in the chat. I am going to share something in the chat as well for you to see how it appears. 

And the best question is going to get a free result from us, from the VWO. Okay. So make a question as interesting as possible, and let’s bamboozle these guys from BAMMBOO with our beautiful questions. Yeah?

So, yeah. That’s all from my side. Over to you, folks.

 

Thierry:

Thank you.

 

Michel:

Yeah. Alright.

 

Thierry:

Before I started with BAMMBOO, I founded several startups. So, before I saw growth hacking as something to work on or to embrace, we adapted it in our own previous startups. We saw, hey, this is like the basic foundation for each startup’s growth. 

And now, several years further, I would like to, besides me, introduce my colleague, Michel Faas, who’s our Head of Growth. So, Michel, please introduce yourself.

 

Michel:

Yeah. Thank you. My name actually doesn’t very often get pronounced correctly, so I was very, very happy about that. Yeah, I’m the Head of Growth at BAMMBOO, which means that I am responsible for the domain of knowledge for growth hacking within the team. As a growth agency, we want to make sure that we also do the process of growth hacking right.

I’m the one assisting the other growth leads with that. Other than that, I also run some projects of my own, which means that I’m working with about 4 customers at the same time, usually, helping them with their growth challenges and also implementing the growth approach there. And those projects range a bit. I’m not going to take too much away from you, Thierry, with your next slides. They range a bit from sometimes coaching people, coaching teams on how to do growth hacking more efficiently or how to start using it.

Sometimes we take over the complete growth process for teams and do it for them. So it’s a bit of a mix between that. So that’s a little bit about me. Back to you then, Thierry.

 

Thierry:

Yep. Thanks, Mitch. So a really short introduction to what we do? Yeah, we are a growth agency. If you’re familiar with working with one before or a growth hacker or a complete growth team, that would be great. What we do is we help companies with their ambitions from a complete, integral, data-driven approach. 

I think that’s really important, not just the traction data, which is of course essential for the top of the funnel, the entire user journey, but also from product analytics and the complete customer journey. I think it’s really important to mention here. So it starts with validation, new propositions, or certain features of parts of a product, or in the narrative, to scaling complete propositions. So it’s a mix of data, tactics, tech, and, of course, creativity. Don’t forget that one. 

As Michell has mentioned a bit already, it starts with supporting teams, a good strategy, tactics for breakfast, all based around data. And then it’s about executing. We do that through sprints, and what we think is really important too is to assure the knowledge within the teams and the companies. It’s not always the dependency on us as an external party.

We think it’s really important to embrace that. Therefore, we have a complete program for growth teams. That’s not just for marketers, but also for all the specialists around them, including product-driven colleagues like data specialists, product owners, etc. We have a program for that called ‘Muscles of Growth,’ and in the future, we would like to share more about it. This is a range of our clients; some of them are Dutch startups and some are international.

It’s a wide range of companies, from innovative startups to, let me say, bold corporations. What do we mean by bold corporates? Yeah, figure it out yourself. Those who would like to not only avoid risks. I think that’s most important. Hey, someone who looks familiar. This is Michel, and this is literally how he sits. So you get a unique perspective on the way it is right now. Perfect for work but also for a gaming environment. Right, Michel? Exactly. Make it as comfortable as possible.

What we do is we have a strategic approach. We always start with a strategic process. I think it’s really important to establish a growth strategy or a growth strategy. And that’s what we call a jump start.

That’s what we start with, and then we execute with sprints. And this is a nice bridge to, of course, experiments because sprints are all about experiments and learning. Not only picking your area of growth with the notorious hockey graph but also testing the bar in the early stage.

So this is the first introduction from us, and now we’re moving along to why rapid experimentation? So this is the first question to the audience. I asked Ajit if this picture was clear to him; it could be a little cultural difference that he didn’t quite understand what the picture is about. To help you a bit here, this is a pancake, or to be more precise, this is your first pancake. Now my question to the audience, and please use the chat for this: I would like to hear from you guys. What are the main reasons or the main reason why your first pancake, most of the time, looks like this? Please give me some arguments. 

Oh, and by the way, we won’t go further if nobody shares some information here. I can’t imagine that none of the attendees has baked a pancake before or anything. So maybe, Ajit, you can help them find their way to the chat. What are the main reasons why your first pancake doesn’t work?

 

Michel:

And if you’re not familiar with pancakes, there’s always the first attempt when doing something new or exploring details. So it’s an analogy, of course.

 

Thierry:

Maybe we have to help them a bit, Michel. It’s still testing the waters, right? So let’s say, the amount of butter…

 

Ajit:

Hear me. Sorry, Thierry, we are getting responses here. And a lot of interesting ones. Somebody has written, ‘Angry life and not hot enough.’ A pan isn’t hot enough because we never did it before. Okay, very interesting. Somebody has not made a pancake before. Does not know how to cook it with ingredients, mix, and temperature to cook. So, it seems that plenty of people don’t know how to cook a pancake. Okay. So maybe your analogy does not fit right.

 

Michel:

Well, that’s still fine. If you don’t know how to cook a pancake, there’s a place to get started, you know, and it’s the same with experiments.

 

Thierry:

Exactly. That’s, of course, the metaphor we’re using here. So if it’s about the temperature of the pan, is it about the amount of butter, or the amount of batter, also important, to where you place the key ingredient. So, it is, of course, an easy joke or this analogy to make the bridge to experimentation. Because before you start, if it’s your first pancake, you have the assumption of how to do it, maybe you read the instructions or manual, but at least you have to test the waters and get out of the building, experience how it works just in real life.

And that’s also how it works with doing experiments. So, and get your learnings there. And there’s also a reason why when you’re doing your first launch or whatsoever, don’t put all your eggs in one basket. Don’t put all your advertising budget in one basket. Please test the waters, do it smartly, do it lean, etcetera.

So far in the analogy, thanks for elaborating, Ajit. And why is that? Because we’re all mammals here, and we need to maximize learning. I mean, that’s in our nature. And by doing that, it’s not only for your existing business but also for new propositions.

So, growth is our formula for growth. Growth is learning plus optimization. And if you extrapolate that to what learning is, learning is experimentation plus measurement, and optimization is more selection times improvement. So what does work, what doesn’t work? Therefore, our first poll. Take it away, Ajit. So, Christina, how many experiments do you run in a month?

 

Michel:

Very curious about this with the audience that we have, with a lot of people coming in from VWO as well. I suspect the number to be up there a little bit, but we’ll see.

 

Ajit:

I see that 80% of people have voted. And the number is, slowly reaching 100 Okay. 90% of people have voted for Thierry. Do you want me to pause the poll, or should we wait?

 

Thierry:

Yeah. It’s okay. 

 

Ajit:

Okay. So I’m closing the poll right now.

 

Thierry:

Yeah, interesting. It’s the maturity level, what I want to go looking for. Again, it’s not that you’re better or worse, whatever, than if you do less. It’s important that you do it at all in the first place. So, I think there’s also a nice bridge to our next slide. So, thank you, Ajit.

Yeah, I think I’m sharing. Because why is that so important? The amount of experiments you are doing. We use this graph. We like this one a lot. I think it’s most important to explain this on the axis. It’s about doing the amount of experiments, as you can see when you’re right, and the y-axis is about the return on investment on your experiments. And return on investment doesn’t particularly mean how many customers or the amount of turnover you make. It’s about the learnings and insights you get based on doing new stuff or optimizing stuff. I think it’s most important here. What you see is the graph also has a negative flow, and that’s the gray part, and that’s the room where it’s important to also get those ones.

Yeah. So inconclusive fields because we can’t draw any results out of that on a statistical level. What’s important is the cadence you see here, the flow you see here. It’s paid by doing more experiments and more often, you see that you’re getting better at this. So the culture and the excitement of doing experiments will grow.

And, of course, some definitely won’t work, and maybe there’s like one jackpot there. And that’s when the excitement grows. The most important thing is what you see in the pink, if it’s good to read here, and we got approval to take more risks. This is something we’ve seen with some of our clients recently where we had organized a program to teach them to do it themselves. Not just us teaching them to face and doing it for others, but teaching them to face.

And they took more risks, also from a product angle, not only on the market angle. The message of this slide is by taking more risks and having a higher philosophy in doing your experiments, you will see increasingly compounding growth. And I think that’s the most important thing we learn here by learning from each experiment and doing that faster, then you make a real step in growth, in creating more market share, and creating more exciting customers and a better user flow or better products because, in the end, you can’t growth hack a crappy product.

So it explained. The gray part, inconclusive or field, invalidation is a learning tool. For those who are familiar with the game battleship, who played that in their younger days or maybe still do, it’s a cool board game, and I do it with my kids. I like it a lot. But the analogy, in my opinion, is one of the best for what growth hacking or doing rapid experimentation really is.

You have the assumption that maybe this battleship should be on some coordinates, or let’s say, in the corners of A and B at 12. But if you hear a splash many times instead of a boom, then you know that you have to get out of there because you’re wasting time and money, so to say. Therefore, if things are not working, this is also really important information. Hey, let’s do something on Discord or let’s set up a new type of content or narrative. Hey, there’s no attraction whatsoever, and knowing which one, hey, maybe this doesn’t work for this audience or at this time, etc., etc. I think those learnings are sometimes even as important as the things that work.

And by invalidations, we mean that you have to take more risks. And oh, we think that is important. 

 

Michel:

And one addition to that is that because you are testing things, the worst thing is not an invalidation. The worst thing is an inconclusive experiment, which means you have to repeat the same experiment again, probably because you didn’t change enough, didn’t take enough risk, or you didn’t do your research or your customer simply doesn’t care about what you’re working on at that point. But it leaves a lot of variables for invalidation because you’re doing experimentation on a small scale. So not necessarily implementing a solution immediately. You’re actually reducing the risk that you’re going to have in your company by getting invalidated learnings as well. Just one addition.

 

Thierry:

Yeah, definitely. Please do that. I think it’s really relevant. And this is also a nice bridge in the previous slide. We should also take more risks, not only focus on one point or, for example, make it more a complete funnel or a certain stage of the funnel or include the product in it. So how do you do that? If we have to build a culture for growth, and where learning is cool, and failing fast, you know, fail fast, we know all the clichés from most successful scale-ups or those who went IPO, etcetera, but it’s about failing fast, and it’s not about this. Please. Hey. We need to do something innovative that no brand is Amazon, anything like it.

Can you give examples of other brands that have hopped on this? I mean, it’s a silly joke here. My point is here. I got something. These things happen, a location.

But the thing is this is what we see so many times. I said we work for both corporations, and this is something we see a lot with larger SMEs and corporations. And even scale-ups, we have to potentially become much more political, etcetera. And where’s the energy, the starting point of the company where everybody took risks, and had the excitement there to explore. And how do you keep that into your culture or, maybe, you love him? Maybe you hate him or his company, but one thing Amazon did really well is to have a culture for growth. As you can read here, the success there is a function of how many experiments we do per year, per month, per week, and per day. Being wrong might hurt you a bit, but being slow will kill you. Well, amen to that. So, one is a pro. So we have a short introduction on what is a growth culture, but some pros and cons that we skipped. What does your team look like? Yeah. Sorry. Your team looks like this.

But we will focus on the mindset shortly, and then Michel will take it from there with the process. So please feel free to elaborate, Michel, because you work with a lot of teams. You have a lot of examples or experiences with teams. But what is a growth culture? These are the basic ingredients if you ask us. Rule number 1, is to build a culture for growth, that there is a possibility to learn and not just, what I read out loud from Mr. Bezos.

So, and therefore, you meet the end of the team and your management. There should be a tolerance for insecurity and access to your data. I think this is a real blocker, Michel, what we see a lot with the companies. Hey, of course, we want to be data-driven. Yeah, bring in all the tools. But, oh, by the way, the data is in silos, and this is so frustrating that maybe this is familiar to some of you. But if you want to really learn, make it accessible, isn’t it, Michel?

 

Michel:

Yeah, exactly like that. The access to data is often fragmented, which is a large issue because you often have marketing teams. They have their Google Analytics. You have product teams who sometimes have Mixpanel or something built by themselves to track everything. And then sales has their CRM. But the question in the end becomes, if you are working on growth in the company, do you really want to force yourself to be stuck on marketing metrics, for example? Don’t you much rather look at how well those leads that you’re bringing in are actually also converting into customers? And if they’re staying customers, which is the reason why we want that access to data to be free of barriers of teams and more accessible throughout the company and across the entire journey. One thing that I think is really important on this list as well, often the most important, is the buy-in of the team and the management, actually.

We see the cartoon that Thierry has shown. We see that it happens, and it doesn’t happen. Sometimes it happens a lot. So teams express that they want to do something, but when they have to do it, they go to their management. They talk about it. And in the end, they don’t get the approval, so they can’t really get to experiment with the things they want to experiment on. Now what you really need here is almost a sponsor from the management team that is going to support the growth team in that sense as well. So that’s something to take into account if you’re just setting up a growth team or if you want to set up a growth team.

 

Thierry:

Definitely. Yeah. Therefore, it’s a really important thing. If you need buy-in, always start with a thorough strategy. You, of course, include a C-level or other important buying and then create with them your objectives or OKRs. We prefer to have a really clear horizon where the experiments should be. And don’t put the horse behind the car. What is the expression again? Don’t start experiments without a strategy or clear objectives because then it’s you’re all over the place. People will see a tech saying, hey, we should test that, hey, that’s cool. But is it really relevant? Is it now here or there, or at least just put it on your backlog? So we’ll take it into account later on but don’t get too excited or fall in love too soon with a certain technique. What should be your big bang? For those silver bullets, they don’t exist anymore. That’s something from the past.

But that’s your best bet. If you follow the sequence, it is really important that by doing experiments based on objectives, select the right tooling, which should be taken into account, and also really important, and then collect your learnings. And then see if there’s like, are you validated, invalidated, or even go back to the drawing table. If you see the line here, which comes back to strategy. Next, what does your growth team look like? Well, the most important thing is that they should be as interdisciplinary as possible.

We divided three of them. Yeah, the first one is the low thresholds. Mostly we see it structured as a matrix organization, where ideally the team consists of people from data, sales, marketing, and product, supported by a head of growth. And what we see in these kinds of organizations, is the following cons, like, a focus.

 

Michel:

If I can make one addition to that, Thierry? On the last slide, what it means to be a matrix organization means that the teams are existing. So you have a marketing team, a sales team, a product team, maybe some data people running around, and you basically say, now I want everyone to spend an extra amount of time on growth hacking. So people, you basically give them an additional responsibility here in the low threshold option. Just wanted to add to that.

 

Thierry:

Yep. Oh, there it is. So what we see there, the cons in these kinds of organizations are lack of focus, not top of mind, hey, day-to-day business. I have to keep my score, my targets, etcetera, etcetera. So leads versus learnings, and performance is measured differently. Michel, could you elaborate on this part?

 

Michel:

Yeah. For sure. So we are working with some people who have a team like this. And they often don’t finish their experiments on time. And when you ask them why it’s, yeah, my day-to-day business is too much, there’s too much business going on that we have to take care of. We have targets. We have a marketing manager who’s pushing us to do something or to achieve something. So growth is an extra that I only do when I have time for it. But what you often see is if you do it only when you have time for it, you’re never going to have time for it. So you need to be pushing yourselves, freeing up your team, really freeing up your team here. In order to make this structure work.

And one of the things to look into here is the lead versus learnings and the performance is measured differently. An important thing here is if you work with a team like this, you set up different targets for, for example, a marketing team in total, but the marketer that’s working with the growth team has to do something that’s different from the target from the marketing team. So, for example, the amount of conclusive and inconclusive experiments run or another metric of growth that they’re going to be made accountable for. So it’s important here if you take them out of their day-to-day to actually really take them out of it. They can’t do it at the same time. You have to take them out so they can do it on scheduled amounts of time.

 

Thierry:

So, on your end, the process, it’s easy to set up. Right? Yeah. No actual resources, besides that of growth. And, yeah, we call it the free trial of growth. It’s the easiest way to set it up to start. Yep. Then if you go to a more complex structure, this is a structure where growth teams within departments that we see, especially when people are growing and they have a real understanding of what their verticals all are about. They have established teams for each part of the funnel, from awareness, acquisition, activation, retention, referral, and revenue, and that’s the abbreviation if you’re not familiar with this one. But what we see there as the pitfalls is, it is a really high investment and many resources.

You need for each vertical, you need teams or specialists or whatever. And it’s a risk that teams don’t share each other’s learnings. That’s something we, hey, we should really focus on just one part because this is our responsibility instead of tunnel versus funnel for fishing, we call this. Maybe you should set up certain parts of the funnel, and therefore, not only focus on your awareness part or retention whatsoever. Now that can be a blocker for the other verticals.

Would you like to add something here, Michel?

 

Michel:

Yeah. You usually see this, you actually often see this in a lot of high-growth scale-ups right now. When you look at Miro, for example, Miro has several growth teams that are working on parts of the product or parts of the funnel. I don’t know how they do it exactly, of course, but, I think the challenge in the end is sharing learnings among teams because one of the risks you have here even if you create a growth team, for marketing, top of funnel and, bottom of funnel, for example, is that they sometimes do the same experiments. And that’s such a waste of time because one of the teams has already learned something about it.

You have to communicate that with the other team. Another con you could see here, which is good in a way, but it can be bad, is that teams can get sort of competitive with each other if it’s not done correctly. This means that they’re going to hold back their information, and they don’t wanna share it.

 

Thierry:

Yeah. Hey. On the other end, there is more focus, and on the slide of the compounding effect, and you can easily create compounding effects by doing so.

 

Michel:

Yeah. I think if you do this correctly, this method has the highest speed of growth. I’m not gonna say the highest ROI, but if you wanna learn fast, this is the way to go really fast.

 

Thierry:

Yeah. True story. And last, in this, this list, the most encompassed is the dedicated growth team who are fully focused. That pitfall here. Our chances should be to embed learnings within the organization. That’s not some of these things like it. Right, Michel?

 

Michel:

Yeah. What you’re basically doing here as well is, by having one growth team separate, it’s sort of more complex than the one before because if you do the one before, there was more buy and your company becomes a growth company instead of just having a growth team in there. But here you have a real team that looks across all the departments that are on there. They look at sales, marketing, and product, and they think, yeah, it’s the same. It’s part of the same funnel. But what this also means is that the sales marketing of the product team is still separate from the growth team, which means that if the growth team learns something and asks the marketing team, for example, to implement their learnings, the marketing teams generally don’t like it too much. So the high risk of resistance, it’s very much there. The same goes for product teams. They offer and have their backlogs for a quarter or half a year. And they’ve locked it in for, say, 90% of their time almost.

If you come to them and you say, yeah. We learned something that needs to change. This has priority because we see it could drive 10% growth in our company. They don’t like it because they have their own plan and you’re there interfering with the things they actually wanna do. So that’s something to take into account here, but there are some benefits.

So, when you extract the growth team from individual departments, you’re essentially shaping a distinct profile within your company. This implies that a growth-oriented individual has to approach things a bit differently than those focused on execution. What I mean by execution is that when you observe the sales team, the product team, and the marketing team, they typically adhere to their established processes. They must be proficient in what they’re doing and strive for excellence. While there’s some room for innovation, you generally don’t want these teams to take excessive risks. You want them to adhere to proven processes that yield results. 

Conversely, the role of the growth team involves scrutinizing these processes to determine if they align with the right objectives. They can then introduce changes accordingly. This means you’re not requiring someone to adapt to a new approach suddenly through the introduction of growth hacking; instead, you’re intentionally hiring individuals with a specific skill set and mindset. This dedicated growth team can operate autonomously from other teams, allowing them to conduct their own experiments without diverting resources from daily business operations. Due to this focused dedication, they can maximize their results efficiently. As mentioned earlier, the highest growth potential lies in the second option we discussed, where you have a growth team for each vertical. However, this can be quite expensive and necessitates a complete overhaul of your business. The approach outlined here provides a way to initiate this process without an immediate overhaul of your entire internal structure. It’s a step you can take when you’re commencing with an interdisciplinary team, where individuals collaborate periodically.

 

Thierry:

Hey. This brings us to our second poll. So, Ajit, bring it on. Of course, this is about the culture of your company.

 

Michel:

One interesting thing here is that you can actually have all of the above. It might sound a bit strange, but if you have a growth team within departments, you may also want a specific growth team or a few growth people to stand above that and look at the entirety of the funnel instead of only the singular components. So, it’s an interesting structure that could actually happen.

Ajit:

Michel, I  think we should also let the audience know that if they have any of the observations, and if they don’t see their option, they can also submit their own answer in the chat. 

 

Michel:

Yeah. For sure.

 

Ajit:

Yes. So, you know, we are not getting as many responses as we had expected or we got the last time. It seems Only 50% of people have voted until now. So I don’t know what is stopping people from responding. Should we wait?

 

Michel:

It could be that you don’t have a growth team. If you don’t have a growth team, select others, and we can leave it with that as well.

 

Ajit:

Yeah. If you have a different observation or if the code team isn’t mentioning any of the options, you always have the option to respond in the chat or in the question. You know, you have both these options open.

 

Thierry:

I agree. We should also consider the time. It’s okay. Alright. Let’s examine the answers with curiosity.

 

Ajit:

Alright. Here you go.

 

Michel:

As anticipated, metrics are the easiest way to begin. It’s really nice. Now, I’m curious if you observe the pros and cons we mentioned within your context. That’s interesting to explore. Additionally, some individuals have a dedicated growth team both overall and within specific departments, which is also very positive.

 

Thierry:

Hi, Michel. I’ve checked the time, and we have 20 minutes remaining. We’ve prepared a section that explains the experiment process. However, it seems that some teams may already be familiar with it or are in the process of becoming more mature.

Perhaps we can share this information later. We have an interesting white paper and a presentation where all the steps are outlined thoroughly.

 

Michel:

Agreed. I think it’s best not to delve too deeply into this and just provide a quick overview. We can skip opening the chapters, right?

 

Thierry:

Exactly. That sounds like a good plan.

 

Michel:

The growth process comprises several components, as illustrated here. We’ll go through this quickly, but we can send you the details afterward if you prefer. Your learning goals play a crucial role in shaping the experiment funnel and process. These goals may arise from specific targets, observations, or directives from management indicating areas for improvement. Regardless of the source, they impact the process, but the ten steps remain constant.

Starting with your growth goals, you move into preparation, assessing what needs to be done for the experiment. Hypothesizing follows, where you determine what you aim to learn. For instance, if the goal is to increase customers through a landing page, the hypothesis might be that altering the headline will attract more customers.

An examination of the current situation reveals existing challenges, and you identify key metrics for the stage. With clarity on metrics, you proceed to design the experiment, creating different variations. The setup phase follows, preparing for the experiment to go live. Remember to capture screenshots of the before and after situations, along with any variances.

Yeah. Going back to the buy-in from management we discussed earlier, a picture holds more sway with people than data alone. While I believe data reveals the truth, adding visuals enables you to tell a story instead of just presenting statistics, which may not resonate as much with many. So, once you go live and obtain results, you work on extrapolating learnings from those results.

If something proves successful, the next step is understanding why. This involves asking customers why they made a particular choice or what they think makes one option better. From there, you decide on your new actions, which will either contribute to new learning goals or form the basis for a new experiment hypothesis if you need to repeat the experiment in a different setting. That’s a quick overview of the experimentation process.

We have a B2B example with an IT company. Previously, they primarily sold devices to other companies, but now they intend to shift towards offering services, not just products. This entails providing devices as a service, leasing devices, or managing security and software. In response to a customer request, they need a new website for this transition.

For the next slide, the goals are clear. They aim to shape a new proposition for their services, develop a website to generate leads and validate the business model in its proposed form. When shaping a proposition, testing is essential; building everything without validation is not a guaranteed success. The hypothesis is that a new website will enable them to test their service-based proposition with clients. However, a critical consideration is whether a new website is always necessary for testing a new proposition.

One significant assumption is that their target audience desires IT services from them, indicating a willingness to engage in both device and service dependence. The assumption also suggests that this arrangement will provide additional value. Testing will explore various angles to determine the most effective ways to engage with their audience. Always keeping in mind if a new website is truly necessary is an interesting question in this context.

Yep. So Moving one step beyond this, we define some actions. We have, like, we’re an agency. We have customer needs, and we have decisions to make we’re going to build the core of the new website, which means we’re not going to build 50 pages and all the blog content and everything like that. We’re going to build what was needed in order to get the website live and to test the proposition.

We’re going to add 3 landing pages to that. To test 3 different angles we have in mind. We’re going to create advertisements leading to these pages and some channels with Google on LinkedIn in this case. Google will be for the pool marketing approach, which is also a test for keywords and SEO moving on from there. For LinkedIn ads, we can test different creators because we can combine them with visuals to test the tone of voice and see how the push marketing approach is going to differ from the pool marketing approach.

Yep. So moving on to the next slide, you can see the structure of how that looks. So we have this as our experiment design overview, which is three landing pages and different ways of getting people on there. For LinkedIn, we chose the angle of easy versus engaged, which means with easy, we take care of it for you, and with engaged, it’s more like, okay. We will take care of you.

So your job becomes easier, and you still have to do something at that point. So moving on to the next one, here are some very quick examples. We made some landing pages. We made screenshots of the wireframes there. There’s some Dutch language in here, but it’s basic according to a best practice landing page.

And if we move to the next one, you’re going to see some of the Google ads we set up. So we had 6 different types of campaigns. We also tried dynamic search. I’m not going too deep into marketing tactics, but the difference is that normally you have to say which keywords to search for with dynamic search, Google looks at your page to decide the best keywords for you as well. Now if we look at the results from this, There are some question marks, some, check marks, and some stop signs.

On the initial landing page, we didn’t see many sessions, keeping in mind the company operates in not-too-large markets. Despite experimenting with small numbers, we did notice that product-first and service-first approaches were gaining traction. Users spent time on the page, engaging by scrolling down and actively participating. However, when we combined product and service offerings simultaneously, the response wasn’t as favorable. Most visitors clicked away, resulting in lower volume.

For now, we’ve decided to focus on separate experiments, emphasizing product-first and service-first approaches. Notably, the product-first approach yielded better returns on Google Ads, while the service-first approach performed well on LinkedIn ads. Interestingly, LinkedIn ads for engagement outperformed ease. In the upcoming experiment, we plan to enhance the landing pages for both propositions, improve Google Ads for the product-first strategy, and enhance LinkedIn ads for the service-first approach.

Dynamic didn’t perform as expected, primarily due to low search volumes, a factor beyond our control when dealing with Google. Drawing insights from this, the product-first approach shows potential through Google but requires refinement. The cost of acquisition remained relatively high, posing a risk of attracting consumer traffic in a business-to-business context. This emphasizes the need for improvement to better align with the company’s target audience.

I might use the same keywords as a company that’s looking for new phones for their employees. You’re going to use it. So there is a risk thereof drawing in consumer traffic. We have to find some ways to sort of filter them out so we don’t have to pay for that type of keywords. The service first does all right through LinkedIn.

I said all right because it performs slightly above the benchmark, but I usually want to go above that LinkedIn has a benchmark of between 0.3 and 0.5 click-through rate. We were on 0-point 51 or something. So it’s not anything too special yet when they approve that. The combination performs poorly on all channels, there’s a low volume of search, going to drop this focus for now. Like I said, for now, doesn’t mean it will never work.

There are no conversions yet. After gathering data, we’ve decided to double the current budget. Additionally, we’re working on lowering the threshold for sign-ups, considering the creation of a lead magnet to encourage user engagement and data submission. This software conversion could potentially attract new leads. This decision is part of the process of selecting which experiment to undertake.

When dealing with very low conversion rates on a landing page, there are typically two possible reasons. Either the user interface is poor, and users don’t understand what’s expected, or the proposition isn’t resonating with the customer at that moment. In such cases, adjusting how the proposition is shaped becomes necessary. Another possibility is that the thresholds for user actions are too high. While the proposition may be interesting, it might not be compelling enough for users to sign up for a demo. They might prefer a free trial or a lead magnet, such as a white paper.

Regarding Dynamic search, its underperformance requires investigation, and we’re setting up a new experiment for that. This provides a brief overview of conducting a business experiment. One key takeaway when setting up experiments is that doing them in isolation can be challenging. While improving a landing page is one aspect, understanding the broader context and user experience is crucial for comprehensive experimentation.

That’s how you can use VWO, to change only the landing page. But if you’re creating something new, always keep in mind you have to send traffic to that, and the traffic needs to be relevant, which means your experiment is becoming larger and larger with every step, every complexity that’s in there that you have to measure both separately so you can see if the channels are performing well, but also in its entirety to see if the entire funnel is performing. Now Thierry has a B2C example.

 

Thierry:

Yeah. But, looking at the time, I prefer it because the B2C example is really cool. I can also send it afterward. It has some similarities with this one, but more from an audience kind of perspective. But I think it’s more inspiring to finalize with the VWO tool.

Yeah. So, therefore, Ajit, if you can, switch to initial as a presenter, and he can finalize the demo from you guys.

From my experience, tools are really important, but how can setting up experiments, more convenient, and the lower threshold to it? And tools like VWO are perfect examples of that.

 

Michel:

Let’s see if I’m sharing the correct screen. Here, I am. And I’m going to do it like that. So can everyone see my screen now?

So what we have here, some of you are already VWO users, and some of you are not. We’re going to show you very quickly how the tool looks and how you could use it in setting up your experiments. So I’ve set up a quick trial account from VWO.

If you’re not using it, you can always do this on their website to see how it looks and how it works. Now, what you see here is the dashboard. I’m going to take you through some steps for setting up a correct experiment in the correct way. Now, I usually prefer if the learning goals come in through a higher-level goal.

But for now, because it’s easier to look at, I’m going to do that through observation. So we have a landing page from us at BAMMBOO. We have VWO snippets. There’s an observation mode with which I can say, well, I want to change things a little bit. Instead of saying hi.

Let’s see. Book a demo instead. So I’m going to save this. I could add a label to make it more clear. But what you can see now, if I go to the observations, there suddenly is an observation for me.

So I had an idea. Book a demo instead, I didn’t have to work with it yet, I can just see how it works. If someone else looks at it, they also see the snapshot so they can see, oh, this is what needs to be changed. So I can give this to the designer already and say, okay. Can you create something for this?

Another thing I could do is there’s no hypothesis associated yet, as you see. Now, with the experiment design process, what you often want to do, I can say there’s a hypothesis based on the book of demo. Let’s say I expect that if we change “say hi” to “book a demo,” we will get higher-quality leads. Let’s say it like that. Like I said, I’m pretty sure this is gonna work. This is always, of course, how sure you are.

One rule of thumb that I actually like here is that you can get 3 points just from data, and the other 2 are gut feelings, which also means if you have no data to support your hypothesis, your confidence should not be high. Because you don’t see the behavior that you’re looking for. Importantly, it’s a pretty high-impact landing page and it’s very easy to do because with VWO, I can just change it entirely. I created the hypothesis. It’s on here now.

Let’s say this is selected for testing. And I’m going to go to the testing module to see how I can do it. I’m gonna create an AB test. I’m gonna take this page. And I’m going to say I’m testing the hypothesis I just set up.

So here you can also see the approach that we talked about earlier of setting up all the right things. Now, one thing that you do see here is that it’s just opening the page, and it’s not opening the editor. Let me just click it again because it happened on a separate screen. I thought, but what you can also see is that the VWO snippet is not installed yet. So if you have the snippet installed, you can go through the editor, and you could just change things on the page with a visual editor.

Now we have some VWO accounts working that I could show to you, but they’re from clients, and I don’t want to put their information out there in the webinar. So I’m going to end the setup for things here. But one great thing here to look into as well is that after setting up your experiments, you can assign goals, which is, well, what I want to see is that when we have the website from us, and people click on say, hi. They go to the contact page and they do something there. Let’s see.

I want to see the click-through rates so I can see page visits to that page. I can also change it. I can also track the button because it could be a bit better. I can also set up custom conversions and do forms, etcetera. So that’s all very interesting to take into account when you’re setting up experiments.

 

Thierry:

If I may elaborate, on what I think is really cool, in this tool as well. It’s also after the login. Right? It’s not only on the front page of your website. So in your product in your onboarding flow whatsoever. You can edit the steps there well and test it. I think that’s what I’m saying.

 

Michel:

That’s one of the primary reasons. One of our customers is using VWO specifically. And that’s to say there’s a login button here that can go into the editor. I can log in. I can change back to the editor, and I can change my product, which is very nice because you can change parts of your product without involving your development team, which is, of course, nice.

One more thing to look into is why I think signing up for a trial in VWO is interesting as well as there’s also an idea log here. So there’s a gallery saying you can change different trial lengths. You can change different buttons. You can change security logos and everything like that. As you can see, there’s quite an extensive list here of sample experiments you can set up pretty easily.

Some case studies of combined experiments that you can look into with resources and articles. So only for this on its own already, I think it’s very interesting to take a look and see how this would happen, how you could use this.

I see it’s half past one now, which means we are at the time limit. I’m wondering at this point, should we do Q/A, Ajit, or what’s the idea there?

 

Ajit:

Appreciate you guys because I see that, you know, out of a hundred people, out of 100% of people who joined the webinar at the beginning of the session, nearly eight people saw them are still online. Okay. So you are clearly doing something engaging, which is holding the people. Okay. So I think we can still continue with the presentation and cover the remaining topics if you want to, but if we don’t want to call that now, we can move on to the Q&A and get questions from the audience.

 

Michel:

Okay. What do you wanna do, Thierry? We can also ask them if they still want to see the B2C case.

 

Thierry:

Yeah. Or more in respect to everybody’s time I would say let’s move to the Q/A

 

Ajit:

Okay. So, folks, I hope you enjoyed it, though, but you created it because I see that 80% of people are still online and listening to the session. So if you have any questions for Thierry or Michel, please feel free to ask. Like I said at the beginning of the session, the best question is going to get free business from VWO. To your address today, after the session itself. So, yes, go ahead.