Insights Over Assumptions: Beyond the Homepage at Pizza Hut

NOTE: This is a raw transcript and contains grammatical errors. The curated transcript will be uploaded soon.

Great. So welcome everybody to VWO Convert twenty five. So I’m really delighted to do another webinar, and this time it’s with the amazing Fabian. So we’re gonna start with introductions really. So you can introduce yourself.

Yeah, absolutely. So my name is Feyden Alvarez. I am the Global Head of CRO and Experimentation at Pizza Hut. And I run all our AB testing, our NBT, our web personalization. We do that across fifteen markets. In total, Pizza Hut has about one hundred and ten markets that we operate in.

And yeah, we just experiment and grow performance as much as we can.

Amazing. And I’m Chris Gibbons. I’m the Chief Experimentation Officer at CredoCX and we’re a dedicated experimentation consultancy. We do everything around experimentation, which includes from all the UX research, uncovering opportunities, all the way to to be building AB tests. We’re delighted to work for Pizza Hut and work with Fabian closely on all of that fantastic fun stuff.

And then as a consultancy, another big part of us is that we enable our clients to improve how they experiment. So we we spend a lot of time enabling clients, building programs, helping them with their operating models, helping to embed service side testing into product teams, all that amazing stuff. Because ultimately, to make the biggest impact isn’t just about delivering work and delivering results, it’s actually enabling our clients to do the same for themselves. That’s a big part of what we do.

So to move on a little a bit more about Fabian for those who don’t know you yet, some good highlights here as well.

Yeah, absolutely. So yes, I was in an agency CRO division a long time ago now. I’ve been in experimentation for over fifteen years across a range of companies, telco, finance, data services, working across B2B, B2C companies as well.

And yes, I used to be a writer for eConsultancy, SEO MOS, have been featured on Glingbo Businessweek on experimentation and CRO and importance of growth at companies as well. I think, yeah, that that you’ve taken me back some down memory lane, Chris. I’ll put it that way. But, yeah, I’ve done thousands of experiments and seen double digit results in single experiments all the way up to the amount of tests that we scaled up as well.

Fantastic stuff. And you still love the area then? You haven’t got bored yet?

Yeah, not as yet, Chris, not as yet.

Absolutely love it. And I think it’s it’s so fun. It’s such a fun discipline to to get into. We’ll we’ll definitely tap into that today as well. Okay.

Amazing. So I’ve got a few questions for you, and we’ll have a we’ll have a discussion around these topics.

One is how does experimentation work at Pizza Hut in a world of franchisees and food delivery, which is a very different world to a lot of the more typical experimentation CRO programs that other people run?

Yeah, absolutely. So I think first, that experimentation, just so you guys understand that I was set up. So we have a centralized experimentation team. We’ve got about one hundred and ten markets in total.

My team operates in around about fourteen, fifteen markets where we’ve run different experiments, AB tests, personalization, multivariate tests. We’ll even go into machine learning experiments as well with multi armed bandits and things like that. So it’s an array of other experiments, I’d say. And I think in terms of the differences and how we go about it, there are nuances.

I think in the fifteen years that I’ve been experimenting, you’ll start to realize that there are just complete differences when you come into a QSR like Pizza Hut. So one of the things that kind of springs to mind for me is just the operational complexity as well for a business like this. So when we run experiments, you’ve got to think about kind of our operations as well. So it isn’t just about a website and running an experiment there.

It’s also about the chain effect that it has to our stores, to our restaurants, and the impact we can have further down the chain basically. So just to give you an example, Chris, that we could run an experiment on say defaulting by the cross size and what could end up happening is we might get some really good results from an online perspective. But if we change the defaults completely, what we could end up happening I guess is we get to a stage where the restaurants would just not have enough stock, for example, of a certain restaurant.

So all the stores and franchisees set up in in a certain way, basically, to make sure that they can deliver for the amount of demand that comes in. So they would already know what types of pizzas are gonna selected more or less, basically. And then they would then prepare accordingly to that. So our experimentation program could absolutely disrupt all our alterations, in other words.

So there is super high risk, I would say, when we test an experiment for a company like Peter Hunt. It’s something that we’ve got a factory to the extent that we run. So, yeah, operational complexity for me is that, you know, quite a quite a big one. I think the the other ones as well, just to be mindful when you start experimenting on like a global brand pizza hut is just to have an appreciation of the differences in culture.

The different cultures, the different types of decisions that people make in terms of their behaviors. We see complete differences in different markets. So if I was to take a market like India for example, one of the things that we will find is that voucher uptake and voucher usage is at a much higher rate, almost probably double that of different markets. So if I was to take the UK in comparison, voucher uptake would be a massive thing.

So there’s differences in culture within restaurants as well. So you can see the culture of, say, India where tipping might not be a common practice. If you go all the way to the US, tipping is a must. I’ve been on the receiving end of that, Chris, basically.

But there’s differences in culture, in other words, and that translates to our websites. It means that we have to think about the different nuances when we’re creating stuff, when we’re creating experiments. It’s understanding and having an appreciation of language as well, that language is not just people translate and we can then just copy and paste that into our experiments. We actually have native speakers and we double check the language differences as well because there are small nuances that can take place as well.

Yeah, it’s absolutely fascinating. I always remember the learning curve we went through as well, actually, because just that how how you need to cooperate, how you need the coordination, extra level of coordination, not just with all the different countries, but with the franchisees, with with the actual restaurants. It was fascinating for us. And I I remember when you were warning us, you said, well, if we’re gonna do this, we need to we need to coordinate with them because they’re gonna run out the ingredients.

I always remember that story always stayed with me. It was like, blimey. That’s it. We never really had that consideration with other clients before, so I thought that was fascinating.

Yeah, absolutely. Yeah, just touching on that, Chris, well. We could run an experiment. We might get the transactions online, but it’s also then the fulfillment, the delivery side of the the business as well. Right? So we used be able to deliver for our customers and and make sure that they are are happy. So it’s a it’s a complete unity between kind of our restaurants, our delivery, on our online on our website and channels as well.

And of course ironically if you approach it like a normal retail website, just improving conversion rates, just sell, sell, sell, etcetera, you could actually make the experience worse because obviously if things run out more quickly, that will create a worse customer experience for people as well.

I think another moment that’s Absolutely, and I think for us it’s not just about running the experiment for the day of the transaction, let’s call it.

Also about that long term transaction growth, like how are we growing that retention side of it.

So we look at all of that and we factor that into our experiments as well. We try to, where possible, not just look at flat line metrics or have we increased conversion rate, revenue, average order values, but actually what’s happening over time. How are people coming back to the site and are they purchasing again and things like that. But yeah, absolutely. You can see it on this graphic as well that you’ve got on your screen that there are an an array of different pizzas. Right?

So we’ve got

A lot of pizzas in with Oreos baked into them, and you’ve got pizzas with cilantro or coriander for some.

That’s in Japan. And then we’ve got Philips Thailand with a cornflake crust, I think we introduced that in Canada as well. So you’ve got all this variation in culture and diversity, really.

It shows you that one is a fun brand and experimentation is basically at the heart of this company that we experiment with different toppings and pizzas and creations basically. It’s an innovative world. So it ties in nicely to kind of CRO and experimentation as well.

Yeah. All I’m thinking now is what I’m gonna have for lunch for dinner, sorry. It’s making me hungry.

I but we we can’t get these we can’t get these in the UK.

Some of these look incredibly interesting flavors.

Absolutely. I think McDonald’s has done a a really interesting campaign as well where they’re bringing all the world favorites, and it’s you know, where you see what’s there in across the world across pizza. In fact, that we’re in, like, a hundred and ten markets. Have mapped over a hundred and ten different variations there of different types of pizzas, things that are unique. And, yeah, absolutely. There are definitely some that would be interesting to taste and to check out. It just makes the holiday travel bucket list a lot wider, that’s for sure.

A hundred percent.

Amazing. Let’s move on to the next topic now.

So why have you adopted a problems first approach to experimentation?

Mean, Sharush, a problem first approach is something that I’ve embedded in experimentation since I kind of got into it about fifteen years ago. I think when I first started running experiments, I was at a stage where we could just we were generating ideas and in the initial early days, I think within the first two to three months, I started to see kind of that flatline results, basically. I started getting into it. And when you just think about kind of the world of CRO and experimentation, it’s to be quite similar to detective work, is how I see it.

So if you were to watch any crime thriller or crime detective program, the main detective inspector will not reveal who the killer is instantly or would it be the end of the program? So it’s the same thing. There’s a gathering of evidence. There’s a gathering of insight and data.

And I think that’s where the problem first approached, something that we massively believe in alongside yourselves, Chris, really about uncovering the problems. So what’s stopping customers from purchasing? Really uncovering that and spending the time there.

Because for us, I sound throughout my career and not just at Pizza Hut is if we solve problems and we solve that through the right types of testing, we end up seeing like a much better kind of impact on our commercial performance wage experiment. And if you think about it, if there’s a problem with for example, people don’t trust, you know, your your checkouts, there’s no point changing the the the color of of the button or the or the headline. Like, it might have, like, an aesthetic colors improvement and and take nothing away. There’s always room to that.

I think security was the issue on a checkout. And if you start to create a concept that try to target that problem, you’re more likely to increase the conversion rate. And that’s what the principle has been for me when I was on agency, when I was telco and across financial services to QSR and food retailing at Pizza Hut. It’s a principle that just gets us the results, we see the impact, but it’s a fun way of doing it as well. You really are uncovering the struggles that customers face on a daily basis.

Yeah, exactly.

And I put this diagram here just to kind of explain the broad concept for people and that is that actually it’s those deep dives starting with evidence from opportunity discovery research, which is qual and quant, really digging into an understand the problem like Fabien was saying.

And and then you start creating the problem statements. And then this is the key thing. It’s almost the order you do things in before you go into ideation and then the all the hypotheses and central solutions to experiment with. And and I think that broad concept is it’s the but too much of CRO and experimentation is is the kind of horse before the cart mentality.

It’s it’s when they you jump. Everyone wants to jump to the jump to a page and then just try to optimize this page, opt get more people to buy this thing. But, of course, to get more people to buy, you need to uncover what’s stopping them from buying in the first place, which means uncovering what problems, what needs they have, what what opportunities are there. And it’s you gotta go to the root cause, and that’s how you find the biggest gains.

But for for some reason in the evolution of experimentation in CRO, that aspect’s been missed a little bit and people have kind of thought that it’s more around just almost the more random kind of iterating, but at a random small scale, or just relying too much on databases, playbook ideas, and forgetting that every business and all of your customers is completely different.

If you approach Pizza Hut with a playbook of ideas that you’ve just pinched from another site, then the chances are they won’t they won’t work. You gotta really get under the skin of the business model and your unique customer base.

So, yeah, I mean, there is definitely a place in the world for, like, running, like, thousands of experiments and trying to optimize something. But I think if you’ve not gone down that problem first approach in the in the first instance, you are gonna be wasting a lot of time.

But I think there’s also challenges for a lot of people that will be watching this today is not just having the right approach. You can just have the wrong type of leadership teams and the wrong type of people influencing that you have to just come up with lists and reams of ideas. Even in my career, I’ve come across a ton of people that have just said to me, We just want one hundred ideas from you on what we need to test. You’ve got to challenge back on some dialer in a really clever way. But sometimes it’s a balancing act for a lot of people. But the closer we get to this problem first approach, the better the quality of experimentation and the better quality of results you start to see really.

And there’s more and more evidence out there for our clients where we monitor the source of the the idea, so we look in the program metrics, and we can start to see that the win rate is higher for those ideas that originated from from research, from quantum quant.

The important thing is not the other way around, which is when you have an idea and then you go and cherry pick some data to back up your idea, which is that very common kind of filth confirmation bias.

And and I think that that that’s really that difference, people get that confused sometimes. So and and it doesn’t mean that you know, when it comes to finding the solution to a problem, this is when you get really creative, which we’ll touch on in a bit. This is where you can can be as creative as you like. But the number one thing is that you’re gonna be still focused on something kind of quite targeted, which would increase your your chances.

And that nicely goes to the next slide actually. So this is one I always use to illustrate the the difference.

There’s an awful lot of ideation, brainstorming ideation sessions like the left that happened when when teams get together in a room and then suddenly it’s like, let’s optimize the PDP type ideation sessions.

And the problem is that they’re not really basing it on the any data or evidence. They just jump to jump to going to their competitors and just trying to think what looks good and pinching ideas off them, which means, of course, you’re always following your competitors even if you are copying the right things, of course. The other thing is your competitors are probably AB testing, and for all you know, you’re copying the losing variant in in that test, which I’ve seen happened before. So you’re you’re always a few steps behind.

But if you look on the right, the key thing here is that there’s a clear on the screen, there’s a clear problem statement, which which has evidence behind it. And the whole team here, this is one of the Creative CX sessions with with Insure, actually.

Everybody is focused on solving that one problem. Mean, they still have tons of good creative ideas which are then turned into the roadmap, but at least they’re all focused on real customer problems.

It’s like a stark difference.

Yeah. I couldn’t agree more. And I think sometimes ideation sessions can just become a wild pool of just ideas that show as many ideas as you can. And I think what we’ve found works well is just structuring it to that original problem, And you you start to uncover that so much more, basically. I think when you closed, you know, grade it down and grade it in, basically, and and grade it in, you start to to really see, like, the quality of ideas coming through as well.

And what I love also is that leadership can get involved still in deciding what problems to go after in any particular quarter. When you’re in whatever model you’re working on, whether it’s like with web experimentation is federated center of excellence type model where you’re just supporting tons of product teams, still the heads of product can have a say and the chief product officer can have a say in the strategy and which problems this team should tackle this quarter. So it’s a really lovely structured approach that involves senior leadership but in the right way. I think that’s why it’s so good to focus on this in this way. You can decide which problems are in line with the key business KPIs and key business strategy as well.

Moving on to number three, this was we thought it’d be good to show you the way we ran usability testing with you, Fabian, because I think it was quite interesting actually.

And it was it was, it’ll become evident where we’ve used this later on in the experimentation days, but I thought I’d I’d share with everybody the type of work we do. And the, what we did is that we we ran usability testing on the core end to end journey, and to really uncover the problems and opportunities across across the journey, especially on mobile because it’s such a high percentage of users using mobile.

And, also, we’ve professionally recruited. The key thing with usability testing is to recruit really well. And if you’re recruiting for Pizza Hut, you need to refine and recruit. This was it in the US.

People who love pizzas, people who are hungry even is is a good thing to do. And then you can have a real natural session, a very natural session. The session which isn’t one of those usability or user testing sessions, which is show them a page, tell me what you think.

It’s a session where we recruit the right users, and we just want them to to fulfill their hunger needs and and get the right pizza that they really want. And the key thing here is that we were incentivizing them as we normally do with professionally recruited users to match your target audience. But, also, we were giving them money to actually go ahead and buy the pizza, and that is key.

When you give money to someone to actually go and buy, that that’s on their focus. It’s just about the pizza rather than pretending to tell you what they think about the side, for example.

Big differentiation. And and when we say end to end, a lot of the time when people say end to end, they mean from the home page to the confirmation page.

But we go further than that. When we say end to end, we mean actually recruiting the right people who have the right target audience, but then all the way to, in this case, actually waiting for their delivery, ordering with the money, waiting for their delivery, and even taking it as far as eating the pizza and trying the pizza.

And, actually, some of the really interesting opportunities we we uncovered was in the the later part, wasn’t it? And I I thought that was absolutely fascinating. So this is how we always wanna one research that end to end journey. And then we take it further on, diary studies on some companies. We take it forward. We have, like, WhatsApp feedback, like, weeks afterwards and things to kind of see how things people get on depending on the nature of of the research we’re doing. But, yeah, this is a lot of fun.

I mean, this was this was a lot of fun. I mean, it it uncovered so much for us to read something. It it would probably need, like, another session just to, like, talk through all the the insight that came from it. But I think the one thing that really that sparks the mind to me is just the noting of we use DoorDash in the US, especially when we can’t fulfill the delivery ourselves. And some of those orders, I think, that you were done in the user testing, I think what we ended up filing was people were getting the delivery from DoorDash, but they were ordering from Pizza Hut and they were just completely confused why there’s a DoorDash individual kind of just waiting or providing it at the door, right? So you have that side of it, but then also there was a bit of anger from customers, which is actually, would have used my DoorDash account and got X amount of discount whilst I’m eating whatever else.

So although we were looking at a certain problem in the journey, I think it was really important that when you do these kind of exercises, you uncover more things. You uncover more different issues and problems that are just invaluable to kind of grow in total conversion rate.

And think by making it a real natural experience, it’s also how you uncover real problems along the journey. So the key thing with usability testing is you don’t want to do it page by page in isolation.

And we’ve seen I’ve seen so many bad examples of this in the past. You don’t wanna show them some new design page and just get them to tell you what you think of it. They’re they’re a user who wants to buy some pizza. They’re not a designer.

You’re not getting tips from them. And that’s the thing. You just want them as the shopper. And It’s like when we do testing with any of our clients, need to make sure they’re the right target audience.

They really do buy luxury fashion, for example, on a luxury fashion site. Or if it’s insurance, that actually it’s within the within the month window of having to renew their insurance. So they actually are currently looking for insurance at the moment. And I think the other thing is we made sure they’ve never used Pizza Hut before so that they are brand new to the site.

So we haven’t learned to overcome any problems which are there. And that those are just some really important tips for anybody doing usability testing.

And then that brings us on to number four actually, which is the problem first experimentation day.

And so so the problem first experimentation days are something that we do at Creative Six, all kinds of brands, and we do two different flavors of them. We do internal sessions where we to really help with the ideation process, we do loads of them with our clients. And sometimes it’s it’s a great culture building exercise too to get more parts of the business involved in experimentation.

And it also it’s also great for senior leadership to see that you’re doing this type of exercise, that you are basing all your experimentation on real customer problems and which which affect the business.

But also we do external variations of this, which is a a fun variant where actually when a when a when a company is wants to get fresh ideas from from a wider audience, we actually invite other brands in sessions. So so they’re fantastic. And what we did obviously, this was amazing. The one we did with you guys in New York, but we we also did one with John Lewis the other day where we did the same kind of external version for John Lewis, which is amazing. All around the the all around the post purchase experience on John Lewis. Again, getting many different brands in to help solve the problem.

It was it was good good exercise. And this was all about this is the internal variance and some nice pictures there. So, yeah, we started off with really, it gives an opportunity to talk about to introduce all the team, Fabian to do the introductions, etcetera, and then also to do a bit of a talk on experimentation value, which I think I think is really useful because we’re inviting people from around the business. And not not everybody is so close to experimentation.

The second part is is really the most important, I think. It’s making sure there’s a good thirty minutes to kind of go through the actual problem, to show the insights, and to format it around a problem statement, which has which has every insight underneath corner and corner and showing videos, for example, and talking through all the insight and and the data as well around the problem. But and then then we get into the fun areas around ideation, which involves individual ideation methodologies and then working as a group in co creation and prototyping. And and then to make it fun and engaging, we did the whole shark tank thing, which which I thought was great and over over judging.

It was brilliant. There was yeah. I mean, it was a fantastic day, and I think the internal facing version of this really not just a hammer home, brings out cauldron of experimentation, and that’s what you wanna do within any organization is just get everyone on board and that experimentation should just be like a siloed off division. It really needs to involve as many ideas and as many as equal as possible. I think it was invaluable for us, definitely, Chris.

And then we move on to the other variant, was the external variant, which was session in New York, which I thought was great. I couldn’t be there unfortunately.

Had other Yeah, know, missed that.

It was it was such a We did we did miss you at that one. I think the beauty of this, the external version is we had different brands, different companies, and different verticals coming towards solving an idea and a problem. And it was just refreshing to have a different of take on things. What we did find, because we kept the brief more or less the same between internal and external, and I think a lot of the ideas were quite similar in nature, but we did find there was less of a less of a kind of a held back kind of view when it was the external group.

They just felt there was nothing to do with kind of an engineering kind of hold back or business kind of restrictions. It was just almost like they had more freedom with their ideas. We started to generate kind of different experiment kind of ideas coming from the external session as well. So I think the New York event was amazing to see the difference on that, especially with people that were focused on experimentation from different companies.

Yeah, was nice, a really good event.

That’s for sure Chris.

It’s fascinating, isn’t it, that? It’s almost, can, in any business you can get a bit too close, can’t you sometimes? And I think that’s what this allows everyone to do, just people to come in fresh and then to explore completely different So yeah, it’s a great activity. And you can see one of the amazing prototypes in second image here that someone did. People get really creative.

Yeah. You’re absolutely right, Chris. It was great to see like loads of people kind of just not just kind of drawing up something, but actually physically making something.

And you could you could tell they were going for the the the distributor was coming for the first prize anyway. But they were trying to make interactive and and really just articulate the idea as well. And I think sometimes it’s just nice to just go beyond just a a sketch, kind of share a visual, in other words, and and just get everyone around the table kind of tossing up ideas and and, you know, sticking this and that and the word rust. I mean, we we completely trashed the room after that, think, to be But I think, yeah, the insight, the fun of the activity is great. On that, that just naturally spurs up more and more ideas as well, which is great to see.

And I think it’s important for everybody to know that it’s there’s something about getting rid of a laptop or getting rid of your mobile phone a little bit and going back to paper and scissors and tape and pens. There’s and actually just having people were in in the room as long as they’re focused, as long as they’re focused on not a real clear objective to solve that problem. I think there’s just something something about using your hands in that way that inspires people to be more creative than they would otherwise.

Savivi.

So so, yeah, this was this was kind of a summary that we wanted to talk about today, and I think this there’s plenty of evidence out there to say that collaborative ideation actually returns better results. But a key for everybody is that collaborative ideation based up a problem statement, which is which is grounded in real evidence from your users. It’s not a made up problem statement. It’s not what you think is going on.

It’s actually based on real data. I think that’s one of the key. And to achieve this, it doesn’t mean you do more work. A lot of the time in organizations, it’s about and a lot of the work we do, it’s about helping clients with their team structures, their internal teams, their operating models to best reposition the existing resources they have.

Many companies have fantastic UX researchers, but they’re somehow very disconnected from products or from experimentation often. So it’s about reusing who you’ve got and the expertise that you have a lot of the time.

Then when you get them into it, the amazing thing is is that there’s always been a problem in the UX around measuring the value of great UX now and analysis. But if we start to tie things into experimentation, that’s how you do it. Like, the value is in the size of the opportunity they uncover, and then we can measure the benefits through experimentation. So it starts to bring everybody together and starts to just allow everybody to have part a part to play in the impact that you drive.

And then we thought we’d we’d just finish off, really, if there’s any last points you’d like to leave the audience with, Fabienne.

Yeah. I mean, it’s again, this this could probably warrant its own kind of session, really, but I think there’s probably one thing that I’d want everyone to take away is to add this kind of realism to experimentation. That it isn’t just you kind of sign it yourself within your working world to just a website.

If you work for a retail brand, then also think about kind of ordering the products yourself.

One of the things that I did when I worked with a beauty retailer on the agency side was we actually just ordered the products ourselves and saw the whole kind of end to end delivery, ended up taking the products. And I think one of the girls in my team was like, Oh, why don’t you try the lipstick on? Just for a laugh. And I was like, No way.

And I don’t know how long that’s gonna last. And that was almost kind of like one of the objections customers were facing basically. And it just opened up new areas for us to really go after and make sure that it’s really clear to customers that this is how long this product lasts for. Does it match x1z skin?

Do I have any allergies or whatever they be basically? So there was a whole host of things that came out of that. So there’s a realism that I think you’ve got to definitely add to experimentation.

We see at Pizza Hut and I’ll explain that at the very beginning as well for anyone that’s missed out, which is it’s not just kind of a website, it’s the whole flow, it’s the whole cycle of making sure that our stores are ready, our delivery is geared up, it’s all a working kind of network. The same applies to most businesses really. It’s just about bringing out real world college experiences from end to end really is invaluable.

That’s such a good lesson, isn’t it? It’s it’s the end to end. Even if all even if you can only have impact on on experimenting and improving the actual pages of the website journey, but it’s still so important to understand, to walk through the whole end to end yourself and also to do research on end to end and try to try to not get too narrow in your focus, like, with blinkers on as such just around that page. Need to step back to uncover the biggest opportunity.

Been such good advice.

Amazing. Thank you so much, Fabienne. It’s been fun as always talking to and drawing out your insights and and all your vast experience in this area. So been absolute pleasure. Thank you.