Crafting Personalized Journeys at 35,000 Feet

NOTE: This is a raw transcript and contains grammatical errors. The curated transcript will be uploaded soon.

Chris Gibbins: Hello, everyone. Welcome to Canvex 24 by BWO. So I’m really excited to be here today with Vicky. Um, my name is Chris Gibbons.

I’m the Chief Experience Officer at Creative CX, and we’re an experimentation consultancy dedicated to the wonderful world of experimentation and personalization. So, um, Vicky, tell me a little bit about your role and, and a bit of background.

Vicky Routley: Yeah, so I’m Vicky. I am currently the CRO manager at EasyJet. So I work across both our airline business, um, which is the one that kind of most people will know about. And then we also have a holidays business.

So I work across both of those, um, leading our kind of experimentation program, the strategy around that, um, and all of those kinds of great things within the business

Chris Gibbins: Fantastic. And I thought so that, um, everybody knows who we’re talking about, I thought I’d actually bring up the website. So, and this is what we’re talking about. So this is the EasyJet website where you can book your flights, where I find myself quite regularly, actually.

Okay.

Vicky Routley: Yeah, so we have, um, uh, two, we kind of have two separate websites at the moment. So this is our airline website. And then we also have a holidays dedicated website, uh, where people can book their kind of full package holidays from kind of beginning to end hotel bags, et cetera. Everything included in one package.

Chris Gibbins: And on the flight’s website you can see that I’ve, you can see we’ve got a bit of personalization happening here where it’s remembered my last visit and also critically it remembers the, uh, the airport I use most often, which is really handy. Um, I don’t know when I was looking at these. Uh, and then we can see all the prices. One of my favorite features is that you can see the price.

But it is, and, you know, I’m sure everyone’s familiar with booking, booking flights, etc. Amazing. And the holidays, also in the Algarve, I see it’s remembered some

Vicky Routley: you’re planning a trip to Portugal, Chris.

Chris Gibbins: I am, it’s because I like to play golf sometimes, that’s why. As you probably guessed. But this is a bit of a background in terms of the websites which you focus on optimising and experimenting on, isn’t it Vicky?

Vicky Routley: Yeah, exactly that. So we run the program across both, uh, both websites. We kind of have a, an overarching strategy that I look after for both, um, both websites, but then we run kind of our experimentation program. On each of them slightly differently, um, at the moment.

Um, but there’s, there’s definitely a, a hope to bring that more and more close together as we bring the businesses closer together, basically.

Chris Gibbins: And, um, we at Creative CX, we support on the holiday side. So we support the teams on the holiday side. And a little bit of support for Vicky sometimes. So

Vicky Routley: Yeah. Yeah, definitely. Um,

Chris Gibbins: how does experimentation work at EasyJet then?

Vicky Routley: yeah. So we have a kind of a bit of a hybrid team, I would say, uh, easyJet. So we’re sort of slightly what people might imagine of COE type, um, set up, but we’re also probably slightly sort of centralized, um, But we actually form part of our product space. So I sit.

Directly within the head of products remit. So everything that we work on is directly linked to our product team, which is great for us. Um, it means that the product team are directly invested in everything we’re doing, which means when we’re delivering all of these great features, um, the product team take those away. And we can make sure that we’re delivering value for the business kind of as quickly as possible.

Um, they’re also so invested in what we’re doing, which is great. Like by having us in that place means that they kind of are involved in everything we test. And they are very, um, involved in giving the ideas and actually working with me on, you know, they might be wanting to develop a new project, but they really want to make sure that they’re doing the right thing, which is, you know, I think A great space to be in that they have that mindset that they want to test everything. They want to know that we’re delivering value for our customers.

Um, so yeah, it’s, it’s a kind of, it’s a bit of a hybrid setup, I would say.

Chris Gibbins: Yes, so let me bring up because I know there’s a lot of talk in experimentation. We talk about it a lot in terms of the different operating models. So if we, if we look at this, some of you may be aware of the term centralized when it comes to optimization and experimentation. And the centralized team is actually where a lot of people start.

In fact, it’s when it’s a small team that’s separated from all the other units, and then an optimization team will have every piece of resource they need to run, to do a bit of research, to do a bit of design work, to do a bit of kind of, and then to build the client side A B tests and get a few tests out the door. But the centralized model often has problems when it comes to it being misaligned with the rest of the business. So you’ll find out that these product teams will be doing all kinds of things. They’ll be building features and innovating, but they won’t be AB testing.

And of course. All of us in A B testing know how risky that is. Um, so, so that’s why there’s been such a, um, an increase. And I know we’ve, uh, we’ve been helping a lot of clients to build center of excellences and to move from this model to the middle one.

And it’s been quite a lot of talk, I think, of center of excellences out there. And the, the center of excellence model is all about where the, Um, experimentation optimization team becomes more of a support function, more of an enabler. And they, typically what they do, the aim is to try to embed the actual building of the A B tests within the product teams themselves, so they’re much more involved. Um, and And this is how you can really scale.

This is how you can really make sure that you’re totally aligned to the business objectives, to everything the product teams are doing, but it takes a lot of work and, and, and your role as an experimenter or, or an optimizer becomes a bit more of a, an education function. You really have to, and, and almost selling the benefits. I think that some of the product teams and, and spending a lot of time with the more on a training and. And also, a technical point of view, it’s like, it’s when you have to get the testing tool on their apps, for example, if they weren’t doing app testing at the moment, and then decentralize it is very unusual, actually.

The challenge with decentralized, if every team are doing A B testing in their own way, that you end up with inconsistencies, and you can end up with a bit of a, I mean, It can work, but it’s quite rare for that to work. You have the other quick thing about the CRE team to mention. It’s also about governance as well, making sure people are following the same process. So Vicky, it was really, it’s been really interesting to talk to you and find out more about your, the way airlines are structured.

And, and it does sound from what you’re saying, um, but it’s actually EasyJet are kind of a bit in between a centralized and a center of excellence.

Vicky Routley: Yeah, and I think the kind of the way we’re set up as an actual experimentation team, we do have developers, designers, um, researchers all within our team. So we can be very self sufficient. We can go and deliver these things and deliver value really quickly. We’re not reliant on being sort of sucked into that product process, which I think sometimes might slow things down.

We actually, by having. Our own, um, functions. It means we can just go away and deliver and deliver, um, at speed, which is a really important thing for easyJet. We really focus on how can we do testing to get the value out there to our customers as quickly as possible.

And by not having that restriction of also trying to do everything else. Um, we then can actually move a much, much, um, way to pace. Um, and it is, you know, where we come kind of more on the center of excellence side is that’s exactly what you were saying that about sort of informing people and teaching them about, um, experimentation is sort of my role a lot across the business. How do we get people to understand what experimentation can deliver and then how do we help those teams?

Um, work with us to actually kind of create those A B tests and create the best the best benefits we can. Um, I think the other thing that we’ve probably kind of talked about a lot is about how it also means We’re not just really doing CRO in like the small sense. We, we are actually making big change. So being linked to that product team is really important because if we make big page redesigns, for example, then the product team needs to be completely embedded in that processor and help us with how do we then deliver that out the door?

Cause it will make a functional change to on a bigger scale than just kind of a button change, for example.

Chris Gibbins: And that’s such an important point, and that’s, I mean, experimentation is so much more than just small optimizations. And not to belittle optimization, I mean, sometimes the tiniest changes, if they’re the right changes, they’re solving a real problem. Can have a big impact, as you guys know. I know you do this really well.

Um, But it’s important to Those bigger features as bigger innovations is it’s important to have a nice balance between small bets more optimization bets But also the big bets sometimes as well because some of them may come off And you can you can make a real headway get some really big uplifts to your key metrics By exploring that but like you said you need the product team involved to do those bigger things So am I right in saying that you? With those when you want to test the bigger redesigns or features You, you’re the expert in how to experiment, how to A B test around those, those areas.

Is that right? Is that where your team comes in?

Vicky Routley: Yeah, so our team will come in if we if we have a big kind of redesign to sort of try and make sure that we go down the right route about that and look at the research and kind of take it in the right direction to then test the right things. Um, I think, you know, coming before. Looking at optimization like this, I suppose, you know, there could have been a world where the product owners, um, just design based on hunch and then just put it live. Whereas we can actually control it.

Actually, the better way to do this is to test it. And maybe you want to test two variations, right? It’s not always that you’re going to have the right idea straight away. So how do we help them with the design of those elements to actually find the best wins?

Um, And I think it’s great that within our team we have designers that understand experimentation and CRO, which I think sometimes is very different to a product design sometimes. And it’s good to have that level of understanding of being able to kind of actually design. For tests and have that kind of wild card idea that I know we talk about a lot, Chris, when we do ideation, like think outside the box a bit more, um, and and don’t just go straight down the expected route. I suppose would be the right right phrase.

Chris Gibbins: Yeah, it’s such, it’s human nature, isn’t it, to, when you’re trying to solve a problem, when you have an idea, well, um, to just go with the one solution. Because you kind of get attached to it and I know we’re going to speak about a bit about that later on as well but you get attached to a Um, and then that’s why ideation is so important to move away from your first initial A b test idea and to actually consider like wildcard like we do at easyjet Um, and and of course if you have the traffic then you can you can absolutely go for abn tests and have more variations Which gives you that that room for exploration um So I know that, I know the other thing that’s interesting when we talk about operating models and ways of working.

It’s like, who does the development and who does the analysis? Because in the ideal center of excellence, you’ve actually got the, the building of the test happening within the actual Product teams, which is where the AB here is in each of these product teams. And you’ve got the idea coming from the product teams, but in reality, there’s, there’s a lot of, well, there’s a lot of COEs come in all different shapes and sizes, and we’ve worked with a few where actually the COE is still doing all the analysis of tests, but they have given the responsibility of development to the product teams, which is interesting.

So they don’t always. that trust the product teams with the analysis side. Um, uh, but, but they have, they have intentions to slowly move over to kind of handing over to a product analyst or someone, an analyst in those teams. And it’s interesting.

So, so how does it work at the moment with EasyJet in

Vicky Routley: Yeah, so, um, it’s, it’s like, I suppose on the, um, airline side, which is kind of where we’ve got a fully in house team. We have developers and designers, but they are, um, directly linked to a product owner. So although they don’t sit within the squad as like a day to day engineering function, you’d imagine, but they do, um, work with the team. The aim is for them to work specifically with a particular product owner so that they build that relationship and that they’re working on the kind of the area for that product owner all the time, which means they’re then kind of almost part of their team.

It’s just that we’re delivering outside the code base and outside of that kind of release schedule, which would. at EasyJet currently be our blocker if we were to put people into those squads. So we like to kind of, that’s where the mid approach happens, I guess, by them, by having them linked to those product owners and working very closely with them, but also being self sufficient outside of that with their designers and their analysts or within our experimentation team so that we can be self sufficient, like I said earlier. Um, and then, um, But it definitely helps firstly, relationship building, right?

How do the developers work with that product owner and the designers to actually understand where the product’s going? I think it’s very important for a designer in experimentation to understand what’s actually coming down the line on a product perspective, because, you know, it might, it might influence the way that they want to go about doing specific testing. So it’s important that they have that kind of link in to our product owners really directly, I guess.

Chris Gibbins: And that’s exactly the, sometimes the challenge with a centralized model on the bottom left is when, when you don’t have a link, I bet people in the audience have been in that situation where they have a small CRO team and you’re, and you’re doing some great work to kind of get the AB testing going and prove the concept and you’ve got some ideas on the page and then suddenly the product team. Launch a redesign of that page, and you almost need to, you need to remove it from your road map. And that’s the perfect kind of illustration of when it’s not in line with product and one of the dangers of working in a centralized way.

Vicky Routley: Yeah, definitely. And I think I’ve, I’ve been in teams where that has been the case before as well. So I think that was something when I came to easy, I was really pleased to actually CRO was seen as a function within product, right? That’s really important.

Um, and I think it makes a huge difference to what we can deliver and actually the value we can deliver because we’re, we’re kind of aware of everything coming down the line, so nothing comes as a surprise. Um, So yeah, that’s a good starting point for sure.

Chris Gibbins: And that’s why here you’ve, you’ve illustrated it with the, by putting it right in the middle of the product and everything around. Um, can you, can you explain this, this, diagram?

Vicky Routley: Yeah, so I guess what I was trying to get at is kind of, in my mind, I was seeing us as kind of like that central piece of a puzzle, like we are really, sort of, involved in, the rest of the business does generally sit around us, but they do feed into us. Our ideas can come from anywhere in the business and they do know about our team. We, we have a really Sort of great relationship with leadership team and things like that. So it’s really good that we’re seen as a really kind of what’s the, like, I want to say kind of the core of the digital space, anything within digital.

Nine times out of 10 has crossed the experimentation team’s path, or at least been discussed with us, which is amazing because it actually shows that the leadership team are invested in what can we test? We want to test, we want to learn, we want data. And that as someone in this industry is a mate, like a great place to start. We.

We want the business to really see us as that kind of central function and help them and, and how do we actually kind of inform what we do for the rest of product for the rest of digital by using experimentation as that, that kind of starting point that middle kind of really core piece of that puzzle. And I think it has every business driven by revenue, right? And it’s, it’s great that also by being in that team and being with product, as soon as we know that there’s a winner and we have a great understanding of that, it’s really easy to turn that around within our space. We’re always talking to product owners.

They’re in all of our calls about results, which means as soon as something wins. We can take that on, we can hard code it and we can, um, run with that change and get the best benefit out to our customers as quickly as possible. Um, and not kind of delay, delay any value.

Chris Gibbins: So you’re not relying on hundreds, on tons of 100 percent served AB test when it’s done. To keep to keep the site going and so you can you can production lines the winners pretty quickly, which I think is an Sometimes that’s an underestimated um Part of the process actually in terms of how valuable it is to get those relationships worked out so that you can get the winners launched properly into the code base because your most of the tests are still client side aren’t they in terms of the The work that you do.

Vicky Routley: Yeah, so all of our tests at the moment are on the client side. But I think, as you said, we don’t, we never run an AP test at 100%. We will never just take the winner and run it at 100%. You can imagine easyJet as a website, the traffic we get to it, we wouldn’t take that risk of running something via a tool into our website to a hundred percent of traffic.

Um, so we do have to make sure that these things are turned around as quickly as possible. And, um, we’ve had kind of, depending on the size of the change, obviously that does make a difference to how long it takes to build. Um, but usually, um, we can kind of get them productionized within the A month to two months, depending on the size of it, on the airline side, because the product team knew about it. They were invested in it.

We’ve involved their developers, the product developers from the very start of the process. So nothing comes as a surprise. Um, and they can kind of lift and shift what we’ve done very quickly without kind of delaying. The benefit of what we’re delivering, basically

Chris Gibbins: Absolutely. No, no, that’s great. And I know on the holiday side, one of the key factors for us to come and help you was to get that velocity up and actually to drive that. Um, and so what is, can I ask what your velocity is at the moment in terms of A B testing?

Vicky Routley: Yeah. So I think we’re kind of running. Um, we’ve actually so on on the holiday side where we’re running kind of probably 10 tests a month, if not building more than that, but running around 10. Um, and I think probably a key thing to call out is that.

That has been a journey for us this year, right? And I think you’re right, Chris. We, that’s exactly why we, we worked with you guys at CCX to help us get to that velocity. Um, we really wanted to actually start learning faster and turning things around.

Um, and I think if we’re honest at the beginning of the year, kind of early this year, uh, we were probably only producing kind of two or three a month. So to now be at 10 only kind of. Where are we? August seven months later is really impressive for us and we’re really ramping up and delivering the benefit with it, which is amazing to see the change and the step change has been really noticeable across the whole business, which is really great.

Chris Gibbins: great to hear. Yeah. And on the airline side, how is, what is your velocity roughly?

Vicky Routley: Yeah, on the airline we’re probably around, not far off, I think we do about eight to ten a month. Um, the difference on the airline side, I would probably say, is more the size of the test that we’re trying to run on the airline side now. We’ve been doing experimentation on the airline for quite a bit longer, so we’ve sort of, you know, Got less of the kind of small changes. A lot of ours at the moment, and definitely this year have been big kind of page changes or funnel changes, which from a development perspective, take a lot more time and therefore our velocity is sometimes.

slightly less because of that. That’s not, that’s not a negative. They’re the things we should be testing. And it’s great that actually we’re testing those big things, as I said at the beginning.

So I think it’s, it’s a positive, um, and eight to 10 a month for us. I would not say that’s a negative anyway, that could, that’s a really good place to be compared to places I’ve been before.

Chris Gibbins: Cool. No, no, it sounds fantastic. And of course it’s not, um, for everybody out there, it’s not just about velocity. It’s that fine balance between velocity, but it’s the quality is so important.

And it’s, and as a, someone in your shoes, Vicky, I guess it’s that balance, isn’t it? It’s kind of making sure which, which we’ll talk about in a bit about research and things, but it’s making sure that the quality is high enough that you’re taking bigger bets, but not all. It’s easy to increase velocity if you just do tiny little tests,

Vicky Routley: 100%. Yeah. And I think that’s something that since I’ve been at EasyJet, so I’ve been at EasyJet for, well, a year and a half now. And I think when I first came in, there was a lot of, we want to do more.

And I was trying to be really clear to, like, stakeholders and business that, you know, It’s great to do more, but we really need to be doing the right things. So how do we make sure that we’re focusing in the right areas? Because, you know, if someone was to, if you set an experimentation team, a target of deliver 10 tests a month, I know that, you know, I could deliver that easily, but what value do you want from those tests? And that was what I was really trying to get back to the leadership team, um, and make sure that it’s great to have their buy in of experimentation.

So how do we have their buy in and make sure we’re. reporting back on the value that’s delivered and not just showing that we’re kind of a turning wheel and everything’s going and it’s going really fast because that’s not necessarily actually delivering benefit to the business. So, um, yeah, it’s definitely been something that this year has been a key thing at EasyJet. How do we focus on that value we deliver?

Chris Gibbins: No, great. And actually you mentioned a few times around the senior buy in. I think that’s always an interesting subject for people, isn’t it? Because um, And how has it been at EasyJet compared to maybe other places in the past or what’s the what’s the senior buy in like there?

Vicky Routley: Yeah. So the good thing about EasyJet is the buy in from leadership teams is It’s there. It’s really strong at EJ. Our, um, digital director is really invested in our experimentation team.

I think that shows in the fact that this year our team has grown from two people to eight people internally. And then we’ve moved to, um, CCX as an actual kind of experimentation agency to give us that focus. Um, and Ian was kind of, The digital director really kind of backed that change and wanted to push for it, which is great, you know, wanted us to actually grow this function and, and share the value with the rest of the business. Um, and we share a lot of what we do with the rest of the business.

So as you can imagine, easyJet is, it’s not a small company on the airline side. There’s a lot of, there’s a lot of different teams and a lot of different business areas, but we’ve spent a lot of time this year Um, doing things like Lunch and Learns to get buy in from, from other departments, because we have so many sections, um, and actually they could all test and that’s what we want to get across. Um, and it’s great that by having the buy in of leadership and also, you know, up to the kind of, uh, uh, the board level. that they’ve kind of understand.

They see our reports and it’s great that they’re kind of involved and embedded in it. Um, we get ideas from our, um, chief marketing officer, for example, like that, you know, it’s, it is an understood thing across the business, which is really great.

Chris Gibbins: And I think that’s the next no that that’s so important. I know it’s um, there’s a lot of talk about How to influence, how to scale experimentation, um, how to get every team involved in this. Uh, I mean, if you like, um, you and I, Vicky, we know the amazing value that it can deliver. So you almost, you almost want everybody to do it, don’t you?

But it’s, uh, but it’s not always, always that easier. And there are people you need to influence and some, sometimes you need to try to get that senior buy in when you don’t have it already, which can be a challenge. But it’s um, I know there’s a lot of focus on that, but I do think it is really important I think it’s hard to sort of scale experimentation unless you have got someone at senior level, really senior level, who’s bought into it. Um, and, and it’s all about finding the believer, really.

What, what are they going to be interested in? Do they want to? We often talk about experimentation, how it allows a company to ramp up their rate of innovation. So it’s not just about revenue, but also we need to be, we need to have a nice balance.

We need to be trying new things. We can’t just be copying your competitors. Otherwise, you’ll just be chasing their tails all the time. How are you going to get ahead of your competitors unless you experiment in the true sense of the word, which is taking some risks, um, and to solve those problems.

So I think that senior buying is incredibly important. Um, You also mentioned where do the ideas come from because that’s a that’s an interesting subject isn’t it for everybody I think

Vicky Routley: Um, yeah. So our ideas. They, we, um, we’ve done a lot of work to get ideas from anywhere in the business. You don’t, it doesn’t matter where that our idea comes from.

We want all the ideas, like there’s never a bad idea. Um, what we’ve really pushed for is making sure those ideas have data behind them so that there is less of exactly what you just said, Chris, less of that, um, our competitors doing this. So we should do it and more of, I’ve actually seen that this is a problem. I think we should go after it.

And this is my hypothesis. And that’s what we’ve really pushed for. Um, but it can be anyone in the business. And primarily we do still get most of our ideas from the product team.

But as I said, because we’re embedded with them, that’s a really great. thing in a way like they’re, they’re invested in wanting to test everything they’re looking at doing. So they want to make sure that we’re involved with their next big idea, um, that they’re trying to solve for the customer, whether that be well in any space. So whether that’s in our post booking space, um, where customers are trying to self serve or whether that’s in our actual booking funnel to drive business, um, revenue, I guess.

Chris Gibbins: Yeah, no, great. No, it’s really, it’s, it’s a fascinating subject, isn’t it? Because it sounds like the product team are using experimentation as, as risk mitigation as part of it, you know, to test everything they’re doing. But also now they’re doing a bit more kind of, um, focusing on the actual problem, aren’t they?

So it’s almost been inspired by, by user research and analysis and all the tools that you have. Um, so I think, so I think that leaves us on quite nicely to the, to the next subject really, which I know is quite, something quite, quite new, but it’s something that’s very close to our heart as well. We focus a lot on this, as you know, I, I’m a bit of a, um, a stuck record sometimes probably, but,

Vicky Routley: No, I, I, this, this has been a real kind of, um, or I can, as you said, it’s very new to us. We’ve, we’ve only started doing this kind of, I guess, um, in the last couple of months, really like properly, but we really are trying to push the business down this, what is the problem we’re trying to solve? Not just multiple lists of ideas that, you know, how do we actually. start with a problem, a customer problem, very importantly.

Um, and then how do we work on ideas and how we might have 20 or 30 hypothesis off the back of that, right? So it’s how do we actually link all these things together? And I think you guys have definitely really supported us on that when that CCX, and this is now how we’re kind of looking to work. Um,

Chris Gibbins: And just to explain this, the real differences, like you were saying, the traditional way, especially in optimization, CRO, was to just have that long list, but it does encourage a bit of a solution first mentality when you have a just one long list, and then you, And then what we would do is many different ways, a bit like the, uh, there’d be a whole list of questions at the top. Is it above the fold? Is it addressed? And it would have questions like, is it, does it have data to support it?

Or was it, was an idea that came from research or data, but it was still effectively solution first. You’re still almost looking backwards. The other really important thing with that, what used to happen a lot is that people would, um, go and find data. After they’ve had the idea, which is, uh, back to the, um, the filth bias, which is a kind of forcing insights and learnings to a hypothesis, like after the effect, which we’ve spoken about quite a lot, that confirmation bias, um, and, and in some ways the, the old traditional CRO framework almost encouraged that kind of bad behavior.

Because obviously, if you love an idea, you’d want to ramp it up, wouldn’t you? And then you’d go in, you’d go in analytics, or you, uh, uh, kind of any tool really, one of the UX analytics tools, and you would find something. And you can always find something to support your idea. You can always just pick and choose only the bit of data that says, yes, my idea is fantastic, we need to do that next.

And of course, the, the fund, I’m not sure if that, actually, I mean, have you, have you seen that happen before?

Vicky Routley: Yeah. So I think when we, when we, uh, when we first started the program at EasyJet, um, there was kind of a, we, we had a very, we had a very complex prioritization model and it did, you know, when I shared that with the product owners and said, you know, the more data you have, the more likely we are to prioritize your idea that almost created that kind of bad behavior of, here’s my idea. And then they’d find the data to match, like you said, but it might be one person in one video, for example, because it’s how it was that that bias like you talk about, like, They think it’s a great idea.

Let’s go and find the data, which is not the right way of doing it. I think, you know, we don’t then deliver the value, right? Because we’re not actually solving a customer problem. We’re just, um, finding ideas that, you know, aren’t necessarily, uh, going to drive any change to either the business or the customer, if there’s not a problem there in the first place.

Chris Gibbins: Yeah, absolutely. And I think the other thing that’s probably one of the biggest, um, kind of thing, things that are missed by working in that way is every opportunity in other parts of the user journey. But don’t get your attention. So actually what happens is that teams end up focusing on optimizing very page level optimization, optimization, just optimizing the pages they want to optimize effectively.

And actually they’re there, which is often the case. There’s often a much bigger opportunity on a different step of the journey, which no one discovers. And I think one of the reasons is because they’re not doing opportunity discovery, end to end research. So it just stays there for ages.

I bet everybody here and listening goes on these certain websites. I’ve been on many different websites and then there’ll be a, there’ll be an issue, a customer problem there that had be sat there for one or two years and you’ll go back and it’s still there and no one’s discovered it. And that’s. Probably because all the teams are just trying, just focusing on their own favorite ideas on their own favorite parts of the website and not doing that function to actually identify other opportunities.

So, so I think this, this whole problem is first prioritization relies on you doing that user research, but you need to be very open when you’re doing it. You need to explore a full end to end journey. regularly, and then you need to be open, uh, open minded about what comes up and what, what, what is discovered. Um, and I think that’s one of the most important things.

And then you prioritize the size of the problem first, before you get to ideate and come up with the solutions. And I think that’s the key thing, isn’t it? You’ve done the work on the putting the problem statement together. You’ve, you’ve sized it up based on how big an opportunity it is, how many people does it affect if it’s a problem and, and how much research you have.

And then actually you you can The other thing is this is when senior leaders can come in and have a say in what problems to go after that particular quarter Which teams are going to go after the problems? And of course the solutions will be a mixture between optimization ideas and maybe some bigger things as well and some redesigns Which is all fantastic. Um, and then only then do you prioritize the solutions once you’ve done your ideation against them And actually I think it simplifies that whole second stage Uh quite a lot Doesn’t it? I think.

Vicky Routley: Yeah, and I think now we can, by kind of starting with that quarterly, you know, problem statements being the things we prioritize, um, and I think that will really help change the mindset as well, like across the whole business of this is, you know, Not just tiny ideas anymore. Actually, we’re going off to quite big things and it means that then senior leaders can have an influence on what, what problems do we want to solve in the next kind of three months or however long you might want to set it for, um, based on what our, what our business objectives might be, because it’s going to have to ladder up to business at the end of the day.

So how does, which problem statements are really going to help us not only solve the problem, that we know is there for a customer, but also going to support us in our, in our business goals longer term.

Chris Gibbins: Yeah. Yeah. Totally. And I think it’s, it’s, uh, um, this lends itself also very well when you have different product teams as well.

Cause in, in some of the bigger organizations, you’ll have product teams responsible for different parts of the user journey. And I think that this method in our experience with other clients actually really lends itself well for that because like you say, you can assign these problems and these objectives to different teams depending on, on their capacity.

Vicky Routley: Yeah. And I guess, I guess that’s why it works for us because we are, we are set up in that way. We have, um, especially on the airline where things have grown so much, but we do have product teams that, you know, look after specific areas, whether that be the revenue funnel or whether it’s the post booking funnel. So it’s, it’s really good because it also means that those product managers can see that their problems are being solved and how are we going to do that and actually really feed into those, um, those ideation sessions and hypothesis as well.

Chris Gibbins: and if I moved to, um, If I move to this slide, actually, this, uh, explains really the, the whole, uh, process on a, uh, by a timeline. So the time it’s going from left to right. And this is, this explains how things, uh, are really, uh, fit in to, to everything and to the teams and to the ways of working. So you can see here that the, on the top, this is what we call the opportunity discovery research.

And this happens quarterly. So typically this is, um, like the recent ones we’ve done, this involves, uh, usability testing, with your, getting really close to your, to your actual target audience, which is always, for those who have done it, always, um, very revealing and very, and you get some great surprises from it, don’t you? And then I think the critical thing is the analysis as well, which is where you, where you start to build triangulated problem statements, because it’s not just about the quality. That’s not just about usability testing with six or 10 users.

It’s, it’s then you need to quantify the problem. So that’s where we use other tools, uh, like content square, um, and to actually start to quantify and put numbers against how many people is this problem affecting. And this allows us to prioritize the problems

Vicky Routley: And I, I think this is a, you know, it’s always, We definitely try to pull in kind of senior stakeholders into those ideation sessions where we’re going to play back that research because to your point you do find out some really interesting stuff in there and sometimes having those senior stakeholders in there to actually hear a customer’s really struggled with a certain element of the website is really important because we use the website every single day. We do not have the same experience as someone who’s sat on their couch trying to book a flight to go to Paris for the Olympics you It’s a very different experience.

So, actually, to see a customer go through that journey, I think, sometimes really helps, um, kind of, senior leaders to even buy in even more to be able to actually really show a user struggling, um, as, as well. definitely been a good help for us. I think a couple of our sessions have brought out some kind of, it’s painful, like you said, sometimes watching people and you think, oh, that’s so easy in my head, but why are they struggling so much? But that’s what’s so important because we don’t see it in the same way that they do.

Chris Gibbins: and it really, um, It’s, uh, I think it’s, it’s also great for the teams who are going to solve the problems too, because one of the key things in this opportunity discovery research is getting a good understanding for the problem. So for example, if you really don’t understand the problem, it’s quite hard to solve it effectively. And what happens is that teams will go down the road of investing tons of money and tons of resource and tons of time. Into designing and building something that doesn’t actually solve any the problem in the right way Or it doesn’t even solve a problem in the first place and then it’s no surprise But you don’t get a winning ab test at the end of it um, so the more understanding you can get the the greater the probability of you of you solving it and Effectively, you’re the higher the win rates of your whole program and the more impact you can make to the business So it’s incredibly Important this, this step.

Um, Yeah,

Vicky Routley: when, when you say about kind of, um, increasing kind of win rates and things, I think that’s what, you know, we’re still at a very, early stage of running like this, but I, I’m really kind of already can see that, you know, in a year’s time when we can report back on how many of the tests that went through this process, one versus the ones that used to kind of live in the old world of idea, find some data to back up. I think it would be really interesting to actually be able to see though the difference between our win rates when we’ve started to move more to this focused area.

Cause I do, I think you’re right. You know, that’s, that’s exactly. How we increase things like win rate is by focusing on the right problems in the first place. So yeah, it’ll be very interesting when we’ve got enough data behind us to kind of look into that in more detail.

Chris Gibbins: and we’ve done similar things with other clients who we’ve been running this with for a bit longer, and we can definitely see the differences and see the increase in win rates, but every client is different, so fingers crossed we’ll get some amazing things.

Vicky Routley: looking that way at the moment. So

Chris Gibbins: No, no, amazing. Um, I thought next it might be quite interesting to focus on like, because obviously you’re doing amazing work there. Everything sounds, sounds great, you know, really a high velocity on both teams. I mean, my goodness, there’s nearly 20 A B tests a month between holidays and an airline, which is fantastic.

You’ve got senior buy in. Which is great. Not everybody has that. And you’ve got that alignment with product teams and the business units.

So, so what are your current challenges, Vicky?

Vicky Routley: um, I think our, I guess our, our biggest challenge at the moment, it is probably where people. Anywhere in the business can kind of become very invested in improving something and how they think that the best way is to improve that customer experience or that business problem. And then sometimes kind of personal opinion and personal views can then kind of skew what actually comes out in the data and they try to, uh, maybe, um, it, it kind of clouds their perception of what the data is telling them sometimes, if, if that makes sense.

Chris Gibbins: But it’s great that they’re passionate. I mean being passionate and actually about this whole about the customer experience is amazing. That’s what you want, isn’t it?

Vicky Routley: And exactly. And this is, this is kind of a double edged sword, right? Because we want everyone to be really invested in experimentation, and it is amazing. And we, but I think what we need, what we are doing as an experimentation team is then making sure that we are presenting back very clear data and making sure that actually we’re giving the data in, um, in I can’t think of the right kind

Chris Gibbins: Objective way or

Vicky Routley: yes, but yes, exactly that. So that, you know, these are the numbers and that doesn’t mean to say that it wasn’t a good idea, right? I think that’s half the problem. Sometimes it’s, it’s how do we iterate on that?

Because it does, it doesn’t necessarily mean it, it didn’t work. That variation didn’t work. It doesn’t mean that. What the problem that person was trying to solve isn’t there.

And how do we actually just adapt and then do it again? And I think sometimes it’s just, uh, kind of sharing that data back in the best way to make sure that actually there isn’t sort of a bias to say, Oh, but it wasn’t that bad. Um, and so it’s, it’s making sure that we, we do the right thing for the customer basically. Um, and, and take that enthusiasm, but actually, Use it for good because I don’t want to lose that enthusiasm within the business for 100%.

Like, it’s amazing that people want to come with ideas and have such an investment into kind of what it’s going to deliver. And, and people get really, really excited by it. They, you know, we run, um, review calls after tests have run for a long enough period. I’ll run a review call with some stakeholders because they really want to know how things are going.

And for me, that’s great. We want people, um, invested in the program. So it’s just. that balance, I guess.

Chris Gibbins: so to drill in a bit. It’s almost, is it because they’re so attached to an idea? Is it that they, um, They can’t always perceive that it may not work, or is there, or is there a danger of not considering other, other possible ideas?

Vicky Routley: I think it’s probably the latter of that around, um, being so invested in thinking that this is definitely the way to go and not necessarily, um, thinking that there might be other ways to solve the problem there, you know, and that’s where, I mean, it goes back to this, right? If we start actually looking at the problem and then breaking it down, that’s a better way to go rather than look at, I think this is solution to solve this problem. There might be 20 solutions to the problem, which is fine. And that’s actually even better, right?

So, um, I think that’s, that’s the point. the key thing.

Chris Gibbins: Yes, it’s almost the, um, it’s quite interesting, isn’t it? Sometimes, I’ve certainly been in situations where you have to almost bring up early on. So, if this solution, if this A B test, if this solution doesn’t work, What should we do then? And it’s almost, it’s actually quite a useful exercise right at the start to consider all the different scenarios, if it wins, if it’s flat, if it, if it loses, you know, what’s next and, and then it helps you set up BAB tests in the right way to make sure you’re learning maximum amounts as possible from the test as well, because you can set up metrics if it goes the other way so you can learn more.

But I think that’s really interesting. So how have you, um, tried to address that challenge or to, uh, to get people thinking that it may not work and that actually, you know, we need to try multiple ideas. And so what, what’s it been like so far?

Vicky Routley: I think that’s where we’re, we’re kind of getting to is around that having the idea of, you know, like we said at the beginning where we have wild cards, like you don’t necessarily Have to have one idea and hope for the best because we are in a really fortunate position working at EasyJet that we get a lot of traffic to both our holidays website and our airline website. And that gives us the opportunity to test multiple solutions to one hypothesis. And actually then that maybe gets them in the mindset of, Oh, maybe, maybe I should think about this slightly differently. Could, you know, there might be other avenues to go down, but I think it’s, I think people in the teams are most.

definitely getting there. I think it’s just about how we communicate that back and think, how do they think more outside the box? And I think getting them down this route of coming to ideation sessions and doing those sorts of things does that because it puts you in a kind of environment where you can come up with multiple ideas, you’re not just coming up with the one. And that does help to kind of broaden.

The horizons, if that makes sense.

Chris Gibbins: Yeah, so it’s actually the, the structure of the ideation sessions is so important, isn’t it? I know, I mean, I’ve, I’ve been in a few so far with EasyJet and they’ve been full of energy because, well, you’ve got talented people on your team, haven’t you? I think for more different parts. So.

But that structure you’re saying is important and kind of helps to drive the right behaviors because I mean, we’re quite mean in those sessions, aren’t we? Cause we force people to come up with quite a lot of ideas, but we don’t let them get away with coming up with just the one and then that’s it. Um, and I think the other thing that we always have at home is that we only do the ideation sessions where we have really clear problem statements based on real users and real evidence that there’s certainly not an ideation session where you, everyone goes in the room and has a brainstorm and looks at competitors to come up with their ideas.

It’s not like that. That’s it.

Vicky Routley: No. And I think it, by us using our kind of, obviously on the holiday side, you guys support us really well with that kind of user research before we even have these sessions. And I, we’re starting to do that on the airline as well with our internal team. And I think it’s really important that by being able to present that data back before you IDA, it’s really important, but also it doesn’t mean that you’re It means that then we’re presenting back what we’ve already found as opposed to expecting people to go away and find issues, which will always mean that people find the things that they’ve already thought of and go and find the data for that.

So it’s good that we kind of do that in the experimentation team to present the research that then they ideate on, if that makes sense.

Chris Gibbins: And it’s that order, isn’t it? That stops that filth confirmation bias completely,

Vicky Routley: Yes, exactly.

Chris Gibbins: But just the order just from I suppose from you as your experimentation team You have to own that those processes really don’t you have to you’re you’re doing so much more than just kind of Designing and building great experiments your it’s your responsibility to set up I guess the whole program to set up those ideation sessions and to make sure the research is happening, etc Is that an important part of your role? I imagine

Vicky Routley: Yeah. And I think we’ve, we’ve been really pushing this year for our UX team that initially they came in as designers. And I think we’ve really pushed this year to expand their remit and their skill set to actually go and do the research as well because then it, it moves us away from them just designing based on the idea that’s come in. They’re actually the ones that are now going away.

doing the research and all that kind of really great stuff that we need within the team. Um, and then feeding into our developers who also sit in the ideation sessions because developers, stakeholders, designers, everyone will have a very different view on how you might want to solve a problem. And I think that creates a really good discussion. Um, yeah, I think we, we sort of own that own that flow, which, which works quite well.

Chris Gibbins: and that’s such a good point, isn’t it? When you say about getting the developers in those ideation sessions too and involved It’s so important because they will developers see the world in a different way to to you and I don’t don’t know probably and and and they also have uh And and they also have a Great idea of like more innovative approaches when it comes to the development side. What’s possible? We won’t always know what the leading technology or new technological solution is to a problem and they can bring that such a good angle and I think people for People forget that

sometimes how important it is to bring different team members of different skills in these ideation sessions

Vicky Routley: Yeah, for sure. And I think we, I, you know, we, I might, I might think that, oh, I’m not going to even put that idea down because it’s going to be way too technical and way too difficult. But we’ve had a scenario where I have done that in an ideation session and I happened to be in a group with a developer that was like, no, that is possible. We can do that.

So it does, it helps kind of that, um, yeah. Yeah, everyone’s going to have a different view of how something might work. But I think having all of those different teams in the room helps to create the best ideas and not just, um, have bias around, or, but what if we can’t do that? Or, um, it stops that kind of thinking by having lots of different people in the room.

Yeah,

Chris Gibbins: great great stuff. Um Another question, actually, because we haven’t talked about, uh, personalization, because of course we’ve got tons and tons of problems and we’ve got tons of opportunities to solve and the program is really going in terms of A B testing experimentation. So what are your plans for personalization? I mean, also, I know from the research, we do have some very different kind of customer needs coming up, depending on whether they’re like families traveling, whether they’re traveling for business work, whether they’re, um, Up for like holidays or city breaks or just, just a quick cheap flight somewhere.

And, um, so what’s your plan for personalization?

Vicky Routley: So I think we’re, we’re sort of just, we’re trying really, we’re trying this year to do more in the personalization space is probably the really key thing. So, um, a really big focus for us at the moment, as you can imagine, is. our airline business is established. It’s been there for years and years and years.

How do we help people that might be looking for a package actually translate into a package customer and not just book a flight only? Um, and then there’s a very different, customer that might want, a one way flight for business to Milan versus a customer that wants to go on holiday with their family in the middle of August for two weeks to Corfu. So we need to make sure that the way we present our holidays offering is targeted to the right customer base and that is something that we’re really working on this year to how do we personalize the journey for customers that might be looking for a package holiday but also how do we personalize our actual kind of browse spaces our homepage or our um, destination guides to really focus in on that difference in customer type that we have now that we have a holidays business and an airline and how do we bring those two closer together.

Um, and I think personalization is going to be a massive thing in that space. We need to make sure that we’re displaying the right kind of content to the right people at the right time. And how do we split down that, um, what we test based on what people are searching for, for example, things like that.

Chris Gibbins: And I think the key thing you hit upon there is the difference. If there wasn’t a difference in user needs and in motivations and what, a particular group of people need if there’s no difference, there’s no opportunity for personalization. But as we all know people have very different user needs and I think it’s nailing that difference. It’s focusing in on Um, what’s the difference?

What does this need for this type of user need? Um, buried compared to this other user, this user going for a two day break, for example, uh, compared to a family and again, you use research, use research to find out what those differences are. You don’t just skip it. It’s interesting in personalization because a lot of it is very, it’s a lot of people go very solution first and it’s just very tactical and it’s just, just do this, this, this, like a whole list of things that the tool supports.

Um, but again, you need to go back to what the problem is. The problem is this completely different needs of, of, between different groups. And you start there, then the solution is delivered via personalized experience, a different journey, like you were saying. That mindset is incredibly important for personalization.

Um, and I mean, you’re doing some of it already. Like, like the fact that you, the website has a memory at the moment. It remembers my favorite, the, the airport where I travel out of. Um, most often, um, London Gatwick, for example, which again, shortcuts my journey.

So when I go back to EasyJet, it’s some of these things are defaulted, which is a type of personalization. It’s kind of very contextual based personalization. So you’re doing some of them, but it’s, it’s great to hear about your future plans as well.

Vicky Routley: Yeah, definitely. And I think we’ve done a lot of work. We did some, um, testing around kind of recent searches because we know that our customers, um, you know, will come on the website. They might search multiple times for the same thing, but they’re searching, you know, on the train on the way to work and then they’ll go home and be with their partner and they’re searching again and we’ve done some work around it.

To your point, how do we save that in their search? They don’t have to keep entering that data. We make sure that we actually make that a kind of seamless experience for the customer by having that as an easy way to search, because we know that’s what they’ve been looking for previously. Um, and we want to do kind of more of that moving into next year for sure.

Chris Gibbins: And that’s how, um, I mean, I would definitely recommend to people, if you want to start personalization, look into the, um, Look into the session continuation is what we call it, which like picking up from where you left off last time, especially on mobile experiences, that’s such a good avenue to start working on. So if, if users indicate through their actions on the website that they’re doing something or like to see something, the next time they come to your site, or even during that session, you default it on. onto that setting. It might be tabs they interact with, it might be filters if it’s an e commerce site, or that, that type of the memory that personalization can, and it just makes your whole website a bit smarter and actually easier to use then for users.

So yeah, that’s fantastic. Well, great. So I thought we would Wrap up with one last question, and this is um, this is whether you can, what would you advise other people in your, in your shoes, other people in experimentation and CRO teams, who wants to, who wants to really scale up the program, um, and, and drive improvements, um, what, what advice or top tips would, would you share with people?

Vicky Routley: I think, um, I think my biggest thing is get, like, go, go and get that buy in from, from leadership, um, teams. And the more you can share the great things that experimentation is doing within the business, the more you can really actually build that program and get, um, move it forward. You know, the more, the more, if you’ve got senior buy in and they’re interested in what you’re doing, I think that’s a really key thing. Um, Yeah, I think that’s, I think that would definitely be a big thing and it’s something that I’ve noticed at EasyJet that has made the EasyJet CRO program so much easier than maybe places I’ve been before because we have that buy in and people are really interested in what we’re doing and it helps us to scale.

So, um, yeah, that would be one of them, I think.

Chris Gibbins: So it’s definitely worth spending the effort in that and being creative in in how you do that and and I’d say almost start with One advocate try to find somebody in in a seat most senior team and and show them show them examples show them surprising results we’ve done that before and then they It gets them out of the mode of thinking that every idea wins type of things there’s all kinds of methods like that.

Vicky Routley: and I think, you know, we did, I think a lot of places do this, but we, we’ve started doing a which test one sort of towards the end of last year, just to show actually, you know, it was interesting. We did it as a quiz and it’s really interesting to get some people that, you know, just assume that a certain variation would have actually won and been better for the customer when actually in reality it was something totally different. And it really got the business to kind of think, Oh God, yeah, like this is actually really exciting and to kind of get that buy in. Um, so I think that is a really, really key thing to, to help, help grow it.

And it’s always, you know, amazing to get more and more people involved in what you’re doing because you’ll deliver more benefit, I think. By the more people that kind of understand what you’re doing and can can really buy into it.

Chris Gibbins: Great. Um any other top tips?

Vicky Routley: Um, so I think the, the biggest thing as well as about, um, finding a way for your CRO team to be, or your experimentation team to be really closely linked to the product team and make sure that you get real buy in from your product space. Um, we have definitely seen the benefit of being part of that kind of As I said at the very beginning, that centralized team, but also being so intrinsically linked to products so that actually we’re delivering what they need and they’re seeing the benefit within, um, within the, the CRO space, which really helps. Um, and I think it’s a great thing.

The more you can kind of try to steer into that more of that COE mindset and get everyone on board is, it has definitely made a difference for us at I can, it’s the first time I’ve actually had that, that sort of set up and I can 100 percent see that it’s definitely a better way of working. And we can deliver on much bigger things, which is really exciting.

Chris Gibbins: All right, so Vicky, one last question. What, um, what books are you reading currently?

Vicky Routley: Um, I’m actually, this is probably a very common one for a lot of females at the moment. It’s just come out as a film so I’m actually in the middle of reading a lot of Colleen Hoover books. Um, and it ends with us which has just come out in cinema so that prompted me to go back to her book series which. It’s a bit of a, like, relief when you, when you kind of log off from work and you just want to read something that’s totally removed from everything you’ve been doing with your day.

They are definitely a good shout.

Chris Gibbins: Nice. No, that’s it. Um, I, actually the two books I’ve just finished, the two books I’ve finished are, um, Marty Kagan’s Transformed, which is really, I’m a big fan of Marty Kagan, talks a lot about product. Again, it’s important to understand product really well, important to understand that world so that you can really, you know, integrate experimentation into it in the most effective way.

So that’s a great book. And also, um, I’ve been reading, uh, Erin Weigel’s, uh, Design for Impact book, which is, which I’ve almost finished as well. And she gave me a nice signed copy when I was at the last conference, which I was very, which is fantastic. Um, so yeah, those are the two ones, but it’s very work related.

Oh,

Vicky Routley: Yeah,

Chris Gibbins: a, take a, a leaf out of your book, Vicky, and

Vicky Routley: Chris.

Chris Gibbins: off a bit. I that’s a problem I had sometime Think that’s been an amazing session. I’d just like to say a massive thank you. Vicky, it’s so inspiring actually to listen to you and listen to all the great work that you’ve, you’ve achieved at EasyJet.

And also actually just to hear a bit more about, um, just to hear the, the positives of, of such kind of a, an innovative company like EasyJet in terms of, um, their adoption of experimentation, of, about listening to their customers and really driving innovation through, so yeah, it’s really, it’s been a lot of fun talking to you as always

Vicky Routley: yeah, thank you very much. It’s been great.