Personalization at Scale: The 4-Year Journey of BAT's CRO Transformation

NOTE: This is a raw transcript and contains grammatical errors. The curated transcript will be uploaded soon.

Hi, everyone. Welcome to Convex twenty twenty five. I’m Matt Maidman, head of CX at Rio Digital. So we are a digital customer experience consultancy based in the UK, supporting brands with a range of services, including CRO and experimentation, accessibility, user research, and many more. And I’m joined with one of those brands today, BAT, by another Matt, if you wanna introduce us.

Yep. I’m Matthew Hanwell. I’m CRO manager for BAT, and I have the distinct pleasure working with mister Mainment almost every day on optimization testing and all those.

Thank you. So what we thought we’d do is just give you a bit of background and a bit of history on the the kind of partnership to date between Rio and BAT, both in terms of CRO, but also personalization and just kind of how the program’s evolved over time and what our plans are moving forward.

So, yeah, just to get into the kind of timeline, guess, we’ve been working together for what, four and a half years now, since the middle of twenty twenty one. And that the the partnership’s kind of evolved since then in a few different ways, mostly in terms of introducing new brands and new markets within BAT to that to that partnership, but also, I guess, the kind of maturity and elements of personalization that we’ve been using as we go through. So just to dive into that, I guess, the first kind of twelve, eighteen months from our perspective was mostly kind of laying the groundwork for where we are now in terms of personalization. So we started very much optimizing for all, I guess, across Vuze, which is one of the brands within BAT, within the UK and French market. So, yeah, initially, the sort of first twelve to eighteen months mostly on the UK and French market on Vuze, so one of the brands within BAT.

And the focus was very much on optimizing the experience for all primarily through kind of traditional CRO and experimentation. I think it’d be interesting to get your thoughts on this, Matt, but I think initially there was some low level kind of segmentation of new and returning audiences, which we’ll go into some examples of a bit later on, and some logged in customers as well, but the majority of it was kind of laying the groundwork for for where we are today.

Yeah. I mean, it very much started right at the beginning of its infancy. So BAT had no previous CRO or optimization experience. We were relatively new to online in truth.

I joined coming from Jet two, obviously, hugely online and everything sold there. I was doing optimization there, but Rio’s joined us at the point where there was nothing happening. So the idea blossomed in the digital team that we wanted to start doing optimizations. We obviously had all the issues that that brings in terms of explaining to people what it is, gathering, buying from people, etcetera. So this kind of journey is from zero to where we are today, I guess. So it’s probably quite useful for anyone that’s having to go on that journey and also anyone working in kinda quite legally tight environments. The nature of the product obviously means that we have to be super cautious with a lot of this thing, but what testing allows us to do is create the best experiences for customers in a fair and legal way as well.

Brio and the guys have been at the very bedrock of all this work. Everything good we’ve done has come off the back of our relationship with Rio in this space, and hopefully, we can talk to you a little bit about how that’s happened and where we are now.

And just working within kind of two regions, I guess, with UK and French markets. Even though the websites are very similar, the product’s almost identical, I think that first year really highlighted the opportunity, I guess, for personalization given the fact that a lot of the tests perform very differently in those markets. And as you say, there’s different kind of legal constraints and and guidelines that we need to stick to. So how’s how’s that process kind of worked?

Yeah. A hundred percent. So obviously, the dream is right that you’ve got multiple markets.

You do a test in one area, and you can roll that exact same thing out across multiple brands, multiple websites, etcetera.

But quickly, that was obviously not the case. Right? We have different languages. We’ve got different cultures.

We have different legal issues that we have to navigate in different places. So what’s happened is we’ve had to test similar things in different areas and tweak quite considerably. I think what’s happened recently is this as we’ve grown the program and proven it out, we’ve brought multiple countries on. So we’ve gone from the two that’s mentioned up to a ton of countries this year, bringing in kind of Spain, Portugal, Greece, etcetera.

So there’s tons there.

But yeah, tough, just trying to find your feet in a world where people have got different levels of experience. No one really knows what this thing is, but the results that we’re delivering look good. As you know, that’s kind of the thing that gets everyone’s attention. Is this making us any money or not?

Yeah, so cool. And then just on the topic of performance, I guess, the programme, just some, I guess, level stats to share.

I think, as you said, the the performance has been great and the the high volume of experiments and sort of campaigns we’ve been doing throughout that period across all of those different regions has given us a huge amount of data. So as you can see here, over a hundred and fifty experiments run that includes kind of the traditional CRO, but also the personalization campaigns.

And the really positive in in that is that huge win rate of of forty percent. And in fact, it’s above forty percent in some regions.

And the from our perspective, think the really good thing is as we’ve increased the complexity and the kind of maturity of the program, that win rate hasn’t dropped. It’s actually got better. So the from what we’ve seen from personalization is because we’re talking, because we’re showing more relevant content to customers, we’re actually seeing better performance from the program the more personalized we get. Even though the audiences of those campaigns tend to be smaller just because of the nature of what we’re doing, the performance of those campaigns has has been really, really strong.

Yeah. Definitely. We all expected a downturn post that kinda, in the horrible phrase, low hanging fruit stuff that we fixed early on, but we seem to have maintained this win rate, if not improved in some areas. So super important to us.

Nature of, I guess, any business and ours specifically is it’s all about returns. Right? So we as much as I love testing and I love personalization, we’re certainly not evangelical about any of this stuff. It’s very much like it are the things that we do it and having visible tangible returns.

And if not, why are we doing them? So it’s a big personal annoyance of mine if people would do personalization just for the sake. Right? It’s about testing personalization, proving it works, and then expanding it.

So as you can see by the scale and scope of what Rio and have delivered so far, we’re really looking into this stuff. So user testing is the bedrock for almost anything alongside these guys as kind of data mining and looking at the issues and checking all those kind of signals of success and failure on the websites.

We’ve got quite a holistic view of good stuff.

Yeah. I think the the customer interviews that you can see there is a really important one, particularly going back to what we were saying about the the different regions and markets because what we’re seeing is actually very similar customer problems that we’re trying to solve across all of those regions because the website and the products are so similar.

But it feels like the way in which we’re solving them, whether that’s because of legal constraints or other things, is is very different. So as well as the test wins, the amount of data that we’re getting from those interviews is is huge and the amount of insight we can take forward. Also just just shows the amount of users that are actually part of a test or a personalization on any one of those websites, over a million users. So there’s just huge amounts of data there that we’ve got to build on now.

Definitely. It also narrows it down for us to, when it comes to user testing, we can narrow the things down to nicotine users, for example. We’ve got no interest as a business of promoting nicotine to anyone that’s not using it currently, etcetera, or anyone that wants come over from smoking, anything like that. It’s all about kinda what do you guys want from us and how can we best serve those people, basically. So we can be really specific with making sure that it’s the right age people. It’s people that are using the products or similar products already.

And yeah, it’s just really helped highlight the kind of information and content we need to provide those people with to make their journey smoother, and as a result, achieve our goals as well.

Yeah. So just in terms of personalization then, what does that mean to as a business then? So I’ve put a couple of goals in here that we’ve talked about overarching goals, I guess, for the programme overall that we’ve got into over the last two years or so.

What does it mean to be AT?

It means what it means to everyone. Right? It’s all about serving the right information at the right time to the right customer, which is obviously a tagline that everyone hears every time we talk about personalization. But what it’s really about is trying to, I guess, get rid of all the noise.

So when it comes to our products, they’re fairly simple in either being vaping or, I guess, modern oral pouch use, etcetera, or our heated products. But there is a lot of flavors, there’s a lot of strengths, and there’s a lot of different levels of, I guess, product maturity. So people in different countries use the products almost. So I guess one of the best examples is products like Velo, for example, is a modern oral product, which is nicotine pouches.

These things are used in Sweden and have been for since before any of us were born, but they’re now spreading in the emerging markets for them like the US and the UK, etcetera, and almost everywhere.

They’re starting to build that base as well now. But those two markets need completely different pieces. Right? So it’s all about how do we get customers to not jump shrimps too quickly? How do we keep you in kind of focused and on the lower side of it all, and how can we help through winter steady use to make sure that the products arrive?

So there’s those elements of it. We have hundreds of flavors, etcetera. How do we show those to the right people, make sure those journeys are simple? And at the base of it all, it’s just how do we make what is a, I guess, recurring purchase easier for customers? And whether that’s on subscription or whether it’s on one time purchases, how can we make our customers’ lives easier in getting a product they want to use on a daily, monthly basis, etcetera?

Yeah, I think we’ve kind of looked at it in three phases, I guess, of the complexity. So as I was saying, I think what we’ve done in the initial kind of one to two years was very much that segmentation phase. So experiences based on device type or whether they’re a new customer to the site, existing, so what kind of content do new customers need that are brand new to the product potentially, or lifecycle stage, so have they been using the product for years and are just looking for new flavours.

That’s very much where we’ve been and I think where we’re up to now is that behavioural element to it, is more around intent and what customers are signalling either when they’re on the site or based on the history that we’ve got about them through the various platforms that we’re using.

So that’s very much where we are at the moment, and to what you were saying around whether that subscription or repurchase, that that is the the focus around that that kind of phase.

And then the third bit, which is where we’re sort of hoping to get to at the start of next year is, I won’t use the phrase one to one, but definitely aiming for user specific experiences tailored towards, again, some of that behavior that they’re showing intent towards online, but also bringing in multiple data sources. So at the moment, we’re really only using the data we’ve got available to us online, that you’ve got whole groups of customers that are only shopping retail and not shopping online at all. So if they do decide to come to the site, account or not, how do we how do we best display content for those customers?

Yeah. Definitely.

The vast majority of our sales are offline.

Just in the same way most nicotine’s consume purchase, guess it’s the standard call at the shop, buy whatever you want to do. We’re obviously in a world where we’d like to provide better pricing and more convenience to those customers using these on a regular basis.

So that’s what all this is about. I guess personalization for us is not about and cannot be about trying to influence customers in terms of you should be having this, or we should be using more of it, buying more, etcetera. That’s not the world that we live in from illegally or just a decency standpoint. It’s all about kinda providing customers the best choices for them, and a lot of this stuff’s based on their browsing signals, etcetera, the things we do online.

Using those signals allows us to make sure that we’re not trying to influence or upsell particularly. We’re just saying these are the things that we’ve got that we we think you basically. So I don’t think personalization’s ever about pillating or tricking anyone. I think it’s just about Showing them what you think they might like, basically.

It’s that simple. Right? I think Yeah. Can overcomplicate personalization a ton. What does Matt want to see today?

And in most instances instances, that is I would like the information so I can make just I want, and then I’d like to get out of here and get on with my day. Yeah. Bigger problems. Fine.

Cool, so we’ll just show a couple of examples just from those first couple of phases of what we’ve been looking at and some of the results and challenges of all of that as we’ve gone through it. So in terms of that phase one, as I said, it’s been very much around that new visitor split, so when we look at the analytics data around the site, the majority, at least, of the online traffic is new. As Matt said, a lot of the majority of sales are happening offline, so when customers do come to the website, one of the primary goals for them is just getting them through to the most relevant category we can, whether that is content or whether that is a kind of starter bundle product. So what we observed through use of the mobile menu initially, and this is going back a couple of years now, high usage of the menu, low uses of search, presented us an opportunity to personalize some of that content for new customers. So everyone saw the same menu.

And going back to Matt’s point, a lot of those options might not be relevant depending where you are in that kind of life cycle. So a few changes that we looked at primarily introducing a link into that starter bundle, which is far more of a kind of educational journey bringing you into to views and educational content around the devices and also some of that new to vaping content. So even if you’ve not purchased offline or online before, what’s some of the content that would be useful if you’re if you’re thinking about making that switch?

And that was super successful. So we saw huge increases in visits through to both of those journeys by prioritizing those at the top of the menu.

And overall, which is kind of the goal of the menu, I guess, is is an increase in visits to products, so two and a half percent uplift in in visits to to products or listing pages, and a really good overall conversion rate up uplift off the back of that, whether that’s through the starter bundle journey or not. So just a really simple example of optimizing something like the menu just for new customers. And then on a larger scale, we’ve also looked at the homepage experience. And this is much more of a kind of guided new visitor journey for customers landing on the site for the first time. So some of the original user research we did and kind of continues to this day, I guess, is is friction around the available product types of different vapes types and flavors.

Just a lack of understanding around some of the naming conventions that are used, and the homepage was very much designed as something a kind of brand experience for customers that are already familiar with Views, and there was no real introductory content.

Again, just kind of assumes you know where to go and and what to shop for, so there was friction there. So what we did was introduce a kind of two step guided experience. The first step that you can see there is basically just us trying to understand are they actually new to the brand or have they shopped offline and they’re just trying to come to the site to order refills, or are they actually brand new? So you can go through that step. And then the second step then starts to showcase some of the most popular and relevant product types and explains what they actually are as well as you go through that.

Yeah, think so I was just gonna mention the thing with this testing, and I think it’s probably applicable to a lot of people, is we spend a lot of time obviously trying to assume who people are, where they’ve come from, etcetera, what they want.

And, obviously, there’s a host of technologies that can do that. But sometimes just ask ask the customer if they want not. What do you need that simple click between those two buttons completely transform journey that we’re gonna put in front of them.

So I think sometimes if you’re getting bogged down and now you can technically deliver something, just take a step back. It’s almost a conversation, right? It’s like, what do you wanna know about today? Let’s and then we’ll provide that information.

Yeah. So Yeah.

Exactly. Yeah. Mean, went through a few iterations of design on that, whether it was, you know, stages of forms and quizzes, but ultimately just that understanding initially of are they new or not was all we really needed there.

And again, really, really well. So we saw a a great reduction in bounce rates specifically for homepage visitors despite the fact that we’re kind of using an overlay to deliver that. So it suggests that that kind of question is is actually positive.

And again, uplifts across conversion rate and and product visits for new customers. And I think the really interesting thing that we can use for future reference in those personalizations is we’re tracking clicks on all of those different elements that customers are using. So on their next visit, we know that they clicked they were new and that they were interested in a specific device type. So we can then use that on a follow-up session. So point of this campaign is you just see it once and we we understand that you’re new, we show you that content. But we can use it in in future campaigns as well.

So that’s been really, really useful for us. And then on the French side, we also looked at device comparison, which, as I said before, just around the the the volume and differences in those devices, there’s been a lot of friction around a new customer just trying to understand what’s available and what’s the best fit for them.

And there was no real comparison content traditionally visible on the website for for any kind of customer. So we introduced an overlay that you can see there, which is only visible to that new customer segment, triggered through a little content slot on on the listing page. If they need help as they’re scrolling, they’ve got an obvious place to go and and trigger that help.

And we saw huge engagement with it. So going back to what I was saying before around the success of some of those smaller campaigns, even though they’re exposed to a smaller volume of customers, over a third of the audience exposed to this campaign interacted with that content slot. So there’s massive intent and a huge signal there that they are looking for that help to compare devices.

And again, similar to the previous campaign, we can then use that intent signal towards a specific device type to follow-up with future campaigns. So even if they don’t go ahead and buy that device on that day or in that session, we know that that’s what they’re most interested in, so we can help them on the next visit as well.

Yeah, and this just becomes more and more important, right? Originally, had a couple of products, comparison wasn’t that important, we were just to innovate.

With the rise of disposables a couple of years ago and now on its decline, we’ve had multiple iterations of products, etcetera. And there are key differences within them, but they’re not super simple to see at first look. So it’s a thing that we had to bring in just to kinda keep up with the business and the changes it was making. And as Matt says, it’s super successful. It’s not rocket science, right? Explain your products a little bit better and people buy it because it’s a little clearer. But yeah, again, a problem that kinda landed in his lap were quickly solved by Matt and the guys.

Yeah. And it’s a good one where I think now we’re looking at what if that’s a great experience for new customers, what is that returning customer comparison? Do they need a comparison between the devices or is it more about introducing content for managing the device they’ve got or new devices, etcetera.

So there’s kind of different Yeah.

And what’s the product journey as well, Matt, I guess? So we obviously know that we’ve got some, I guess, more expensive devices, like with our Beuzel that’s come out recently, and then we’ve got some much cheaper devices like these reloadables.

They’re completely different price point. One’s thirty five, forty euros, the other’s five to ten, etcetera.

But both do a similar thing, but once you get the products in your hand, you can feel the difference and it is more superior. But what we get is customers go through the products as well, right? You might come in on one of the cheaper end up at that premium thing, so it’s not just a short journey, it’s like how long customers have a reload, for example, before they upgrade to one of the Ultra devices, etcetera, and what’s the benefit ad we can give them. And a lot of that has to be experiential within the websites as well as using the product itself.

So again, I guess trying to look at when they’re gonna make the move and what information they’d need to what we believe is best experience we can deliver from a vaping product, which is the culture product that’s live right now.

But then again, cautious that that price point just might be too high for some consumers. So this is why we have these other products as well. So how do we show the right things to the right people again? Because again, we’re the personalization for us. Let’s not put people off with higher pricing and premium products, but those people that want products, let’s not put them off without with not showing right. So it’s that constant balance between getting the right stuff for people.

Yeah, exactly. And that is a nice segue into phase two, which is around Ultra at the moment. So couple of examples of what we’ve been looking at here, and again, as Matt said, this is very much for aimed at customers that have purchased the Ultra device very recently, the new device that’s launched, and what kind of content do those customers need. So the first experience of Ultra consumers, so customers that have already bought the device, was around device registration.

So there’s a couple of benefits to to promoting this. The first one is there are a load of benefits you get just by registering your device online as a used customer, including warranty and other and other bits. But there’s also a benefit to the the data that we’ve got, so we can we can help target those customers if we’ve got that device registered on the website. So one of the first campaigns we looked at was a homepage takeover that you can see there that appears on that first visit post purchase of the device, talking about the benefits of signing up.

And that that that’s live at the moment, we don’t have any firm data on it, but we are seeing again high engagement with that content.

If you look at the the the current website experience, it’s probably aimed at customers that haven’t yet bought that device, So there’s no point promoting it to them if we know that they’ve already bought it. So let’s let’s give them some supporting content to use. The second one in that phase two is around the transition between the pod types. So as Matt said, there’s a huge number of different types of pods at different price points.

And for customers that we know have already bought the Ultra device, it makes sense to show them the product that gives them the most benefits to the device they’ve already purchased. So the Ultra pod has various benefits to it, specific to Ultra compatible with all but most of the benefits you get with that device. So talking about the fact that there is a compatible pod for your Ultra device on the homepage makes a lot of sense. So again, bringing in some of that previous purchase history and activity and intent on the website to to navigate customers.

And then finally, the product sorting. So this is almost the reverse, which is for customers that haven’t yet bought the Ultra device or bought a different device, how do we update and amend the sort order of products on listing page to to be most relevant to customers? So as you can see there, there’s a huge number of of products through the listing page, particularly on mobile. There’s a lot of scroll effort to go through to to get there.

By default, most of the UltraPods are shown at the very top of that list.

But obviously, it’s only really relevant for customers who have already bought the device. So for those that haven’t, do we show them the the products that are relevant? So again, similar to kind of product recommendations and just the general experience of landing on the site, what products are we trying to prioritise based on what we already know about you to make it most relevant? And then as we said, phase three is very much about twenty twenty six. So what are kind of your existing challenges, opportunities you’re looking at for twenty twenty six, Matt?

I think there’s a couple of challenges as the portfolio grows, ease of explaining everything online, which we’ve already touched on, obviously that gets bigger.

As the price points vary from quite low to quite high, as we’ve mentioned, what to show customers, that could be a problem, it’s not we don’t wanna put anyone off, but equally we don’t want to stop people buying the most expensive products, right?

How do we garner loyalty in quite a competitive space, so there’s multiple brands. There’s thousands of different vapes, etcetera. What sets us apart and how do we explain that to customers? Believe it’s the quality, etcetera. All the thing that we can deliver is a is a huge vaping against some of the smaller guys which are putting things together, etcetera. We think that’s experiential things are different. Things like having an app, things that come along with being an ultra member like events, etcetera, and all those pieces that you come around.

Twenty twenty six for us is very much be about doing more of this good stuff. So trying to get the right things into the right people, trying to identify those people sooner, and I guess nurturing them through, all while keeping ROI as high as possible. That’s the real goal. We just believe that this next step of personalization, what takes us over that threshold into doing good stuff to great stuff. So on a personal level, it’s exciting because we want to do this kind of stuff. We want to be involved in that.

But realistically, everything’s business based. So it’s testing everything, making sure it pays its own way. Let’s not just get vein around doing some fancy tests that are pointless when it comes to the bottom line, I guess. Yeah, it’s more of the same and getting more technical if becoming more technical, Gan has more business reward, I guess.

Yeah. Yeah. And on the mobile app, I guess at the moment, that is very much a non kind of transactional experience. It’s purely to support the use of the device you’ve already got, essentially.

Exactly.

How do you see that informing the experience online, I guess?

Because that’s a huge amount of data that’s kind of untapped at the moment, I guess, of usage of the Yeah, right.

Yeah, it’s another touch point. That’s one of the main the app is it’s great to use, customers who are using it are enjoying it, There’s a lot of data on there that’s useful to those as consumers.

What that looks like in the long run, I don’t know what we can put on there and what that what can be more useful within the app is yet to be fine defined, I guess.

But we know, like anything else, when we’ve launched something and not tested it, there’ll be a load of issues or improvements that are ripe for doing right. So I’m sure there’ll be some focus in how we improve the app going forward as well.

So the app piece will become big for us And more about how we can talk to offline customers and how we can bring them in and kinda get them converting online, which is obviously the from my point, as a CRO and eComp guy, that’s where we prefer, but also completely accepting it’s not about what we want. Right? It’s like, let customers shop where they wanna shop, and let’s try and give them the most joined up experience we possibly can. So if you’re we know that some people start online.

If you run out of product, probably just gonna pop to the shop. It’s that just because of the nature of the product. It’s I would like it, please. It’s something I do every day with my coffee.

So I have to continue to kinda drive the program forward, I guess. So the right thing for customers and improving the experience for them hopefully will lead to continued success business perspective.

Okay. Thanks for for tuning in everyone. If you’ve got any questions for Matt and I, feel free to ping us a message, but thanks very much for watching.

Appreciate you. Thank you.