From Bricks to Clicks: Primark’s Bold Step into Digital Experimentation

NOTE: This is a raw transcript and contains grammatical errors. The curated transcript will be uploaded soon.

Chris Gibbins: Hello, welcome everyone to the BWO ConvEx 2024. I’m absolutely delighted to be on this, um, call with, uh, with Annette actually from Primark. So, um, just a quick intro. I’m Chris.

I’m from, um, Creative CX. We’re a specialist experimentation consultancy. So we do everything AB testing, UX research, all of that fantastic stuff. And one of our clients is Primark.

So I’m delighted to have Annette here today. So. Yes,

Annette Rowson: you. How are you doing?

Chris Gibbins: really good. Very good. Um, excited to dig into some, a few things more Primark and learn more about what you’re doing over there. So do you want to intro yourself first and what you do?

Annette Rowson: Sure. Um, so I’ve been at Primark for a couple of years now. Um, I was brought on to start experimentation here within digital and digitech. Um, so yeah, I mean, that feels like a long time ago now, a couple of years ago, but we’ve actually launched the program in May, um, which is why we’ve brought in Creative CX.

Um, and yeah, we’re scaling pretty well. Um, and in the midst of that, about a couple of weeks ago, I also brought digital analytics under my wing too. So, we are aligning experimentation and digital analytics as well.

Chris Gibbins: Fantastic. So I think what would be really interesting for people to start off with is to hear a little bit about, The Primark story because you guys weren’t really into doing e commerce until recently until um, So i’m really interested to hear the hear the background

Annette Rowson: Yeah, so, um, Pre Annette’s era in Primark, um, there, um, was a business case put together, um, for Click and Collect. So we are not your traditional e commerce site. Uh, we were doing so well, um, in our bricks and mortars business, our stores are still our main focus, but we wanted to bring in, um, an offering that would, um, aid us in playing within the digital markets. Um, and that was Click and Collect.

Um, we do have a very pretty slide about, um, our timeline. That’s my, that’s my education for Chris to be like, can you please get to the timeline slide? This is our beautiful timeline. Um, so this is our Click and Collect journey so far.

Um, as I said, I’ve, You know, I started experimentation here at Primark. Experimentation has been my personal bread and butter for many years. But Click and Collect, for us as a whole, is actually one of the biggest experiments I’ve ever worked in. Uh, we started in a small kind of sample size of, um, of, um, stores.

that were in the Northwest. Um, and that was just with kids where a nursery and that proved to be really successful. Um, so we actually launched that in the end of 2020. And then, um, in July 2023, so last summer, um, we expanded to all stores within London.

So I’m a London girl, so I was very happy about that. Um, So, um, it is currently in London, um, any, actually, the, the correct term is any store within the M25, it just doesn’t, there isn’t as, as, um, it’s not as memorable as within London. Um, and then, actually, in May, um, this year, it was announced that Click and Collect will be expanding to the whole of Great Britain by the end of 2025. Um, so it has been an experiment in itself.

It’s been a huge experiment and, um, it’s been great to be part of it and seeing all the kind of, um, Pieces of the puzzle that we’ve needed to put together and also see the team grow so rapidly, um, even in experimentation and digital analytics. And we’ve just had, um, a new start. And now we’ve got two more analysts, um, joining and also, um, a specialist, uh, within experimentation joining at the end of the year. So it’s a really exciting time for us, um, here at Primark, um, and I’m really, really looking forward to the expansion.

Chris Gibbins: Fantastic and it’s this is a quasi experiment really isn’t it where you you weren’t able to there’s no way you can properly and um, a randomized A B test, something like as big as this. So that’s why you chose, um, representative areas and compared the performance of different stores to one without in the trial. And of course for click and collect is a very, it’s a perfect kind of omni channel journey, isn’t it? Where it’s not just about the, not just about the online, you have to see what happens to people when they get to store, which is, which is fascinating part of this experiment.

Annette Rowson: Yeah, I think for us. Um, keeping the stores at the forefront is definitely a really important part of our strategy. Um, Click and Collect gets people into the stores. Uh, if you go into Appointments Store and see our Click and Collect offering, you can see that the tools are in a way where Um, you can walk through the, through the store and anything that you kind of missed online, you can actually pick it up, um, install whilst you’re picking up your Click and Collect order.

Um, and it’s, it’s that, um, that additional item in the basket. It’s the second basket that’s, for us, very valuable.

Chris Gibbins: And of course at Primark there are so many easy to add items out there, there’s a lot of, it’s very reasonably priced, so it’s very easy to throw in another item,

Annette Rowson: For me, it’s the phone charger. It gets me every single time, those phone chargers. Phone chargers, you know, the extra wires that you can get. Yeah, they’re very well placed at the till and I was like, Oh, I could do another one of those.

I always end up getting one of those. But yes, um, we are in a market where, um, You know, those items are very accessible for everyone. So, um, yeah, we’re really hoping that that second basket is, um, going to be even more valuable once we’ve expanded nationally.

Chris Gibbins: Yeah, and I think this hits upon a really important point as part of experimentation. Experimentation. is, is almost a mindset. It’s a way of thinking.

It’s a way to get better evidence. It’s a way to get near to, um, we want to detect kind of cause or inference, but we, and sometimes it’s not possible to do everything within your, your A B testing tool. Sometimes you need to be more creative with how you set up the experiment. Such as, um, many, we have other clients that do, that test their advertising, that test the power of ads, but they can’t do that all the time for an A B test, so they have to, um, think about what the sample is for the A and the B version, and test in different areas, test in different parts of Europe, for example.

So there’s, there’s all kinds of, uh, part of the skill of being an experimenter is your creativeness, I think, in setting up the experiment in the first place.

Annette Rowson: I think for us, definitely whilst we are still within a smaller sample size for Click and Collide, I think it’s been a real opportunity for us to actually learn that skill, um, and figure out as well that. with the different markets that we actually can do a B testing in. So there’s a couple of other markets that we can, from an engagement perspective, um, gain some, uh, some wins or some losses, uh, but actually conduct experimentation in, um, but for click and collect, I think that’s one of. For me, selfishly, one of the big reasons why I’m quite excited for the expansion, because it just means that from an experimentation point of view, those opportunities will definitely grow.

Chris Gibbins: which brings us on to the second question that I have here. Where are you on your experimentation and personalization journey at Primark? So you touched on it already, but where are you in terms of like, you said you launched a program in May, uh, you’re in the early days, um, what does that look like?

Annette Rowson: We are definitely at the beginning. I think I spent the first year in my role, um, driving a data driven culture. So getting people to adopt certain tools, making sure that people are instilling that data driven experimentation mindsets. That was sort of my role.

prior to the program beginning, um, we were going through an RFP implementation of the tool, the testing tool, et cetera. Those are the things that kind of prevented us from testing right from when I began. Um, so I think that was, One of the things, so I had to think about, be quite creative of, well, what, what can I do here to, um, create opportunity here at Primark? And I’m quite lucky that I had a year to actually do that.

So it meant that once experimentation was possible, uh, within our environment, we could do that. start, um, we could start testing, but with making sure that there were stakeholders that were on board, um, because we’d already instilled that kind of mindset, we’d already started certain initiatives that, um, that grew that, um, going back to your question, like where we are at the moment, I mean, we’ve just started. So I actually counted before this call, we’ve done 10 tests. Um, we’ve done two server side tests, which is, I think, a success for us, and that means that, um, because at the moment, digital and digitech are, um, are, well, they’re separate, but not separate.

But it just means that cohesively what everybody’s on board. So it means that, you know, we’ve got product owners, we’ve got developers that are on board with this kind of mindset, which is quite difficult considering where we were two years ago, where we were doing this like international migration and it was like a long list of just do it. So it’s like, how do you go from this kind of just do it mindset to, okay, we now need to think about what we’re optimizing, what tests we’re going to run. And.

In what order so the prioritization piece and those two mindsets are actually very very different So i’m lucky that I had that time at my beginning uh of of being here at primark, um But yes 10 tests so far really hoping that we will scale. Um, Really interesting results as well. And I think now We are in a stage where We have to get people used to You The losses. Um, that’s probably the next challenge that we have is, okay, we’ve spent time building this test.

It’s lost. What do we do now? So does everybody feel hard done by, by it? You know, you know, you might have stakeholders that want to dig into the data even more and try and extract a win when it’s actually just a loss.

Um, and it’s just managing those expectations as well. Um, so yeah, so that’s where we’re at the moment. Really exciting.

Chris Gibbins: Good, and it’s a scale up as well, isn’t it? So it’s it’s a lot more in the last month or so than than in in may for example in terms of the scaling the number the tests

Annette Rowson: Yeah, definitely. I think. It’s one of the things that we really want to track. We want to make sure that we’re scaling up, but we also just want to make sure that we are focused on those priorities.

So you can scale up with quick wins with easy to implement tasks, but are they also tests that are going to bring us value? Um, so that’s definitely, um, the juggle there.

Chris Gibbins: And getting two server side tests done is is it’s pretty good You know, there are many organizations, but it takes a couple of years of client side testing before they get to that stage Um, so it sounds like that’s um, and actually that’s because of how you’ve You Started to embed experimentation, isn’t it? But you’re not when it comes to operating model, um, you know, when there’s sometimes a very centralized team where it’s where it’s a small optimization team who are doing everything, including coming up with all the ideas and all the A B tests and everything versus the center of excellence, perhaps when you’ve when you’re actually getting all the product teams involved and you’re starting to become more of an advisor of support function, um, versus Decentralized where every single team is doing experimentation, but often in a different way.

Um, where would you say you are at the moment?

Annette Rowson: I think it’s really interesting because I think the more I am here at Primark, the more I actually think that I am an advisor. I just don’t internally believe that I am one but I do go on calls and think, gosh, I’m advising here not really doing Very much, but it’s, it’s more about instilling best practices and making sure that, for example, the right KPI is being used for success, um, making sure that the prioritization framework is, is definitely aligned. I think in terms of how we’re operating because we’ve just started, I am in the thick of it, definitely. But I think as the team grows, the alignment with specialists, experimentation specialists with each product squad is definitely a vision that we have.

Um, so that will be the plan going forward. And once you have a specialist that’s embedded in a product squad. in terms of service side tests, in terms of understanding what value, but also mitigating risk, those levels as well, that kind of best practice advice, um, the experimentation mindset would solely sit with that specialist. And we found it to be really successful so far because in one way, it, I think at the beginning it was sort of like me with a sales pitch, you know, saying I’m ready to get on board this experimentation mindset train.

But actually once you start putting a structure in place where you’re actually embedding somebody within your team, who’s a specialist with an experimentation background within a product squad, that is probably going to be more successful in terms of scaling within that squad. Um, So we’ve, we’ve, we’ve trialed it with one squad and it’s been really successful. And I’d probably say, I mean, if they were in the room with the product owners in the room, they, they definitely say that in terms of how we’re more aligned, we’re working closer. We have definitely a more harmonious relationship as well.

Um, and, and also it’s a shared roadmap of what that looks like. So, you know, there might be. something that’s aligned with their KPI that I’m completely unaware of. But when we’re collaborating, actually looking at the prioritization framework together, um, the kind of agreement or, um, yeah, the agreement that we have in terms of what server side tests is going to be put live or going to be built.

Um, the alignment is across. Um, and it just means that there’s no kind of miscommunication about it. And it’s been done in a way that is very aligned with experimentation. Best practice.

So I’d probably say that’s how it works at the moment. Um, but I do think that because we are product ways of working here, that was the way that we could actually start scaling. Um,

Chris Gibbins: So, so let’s, let’s dig into that a bit more because that’s quite interesting, isn’t it? So a lot of people out there probably have different roadmaps. So they may have a product because you have three squads, which are effectively three product teams, don’t you? That are responsible for different parts of the journey for different parts.

Cause you have an app, don’t you have an app and you have the, um, The website, um, and you have different teams responsible for different parts. Is that right?

Annette Rowson: yeah. So no app at the moment, but we, um, in terms of the,

Chris Gibbins: sorry, I forgot.

Annette Rowson: um, but at the

Chris Gibbins: Not yet.

Annette Rowson: We have, um, we, there’s more squads, but for experimentation, we align with search and browse, so that’s everything to do with search and PLPs and PDPs. We have content and inspiration. They’re in charge of, um, article pages, homepage, um, category pages. And then we have checkout, which is.

Check out. Um, the plan is, is once I actually only have one specialist in my team at the moment, bless her, who has been part of building this and making this a success. And the fact that we’ve, so successful has warranted another specialist to come on board. And at that point, we will start splitting the specialists into squads.

Um, but even with just the two of us to date, we have a prioritization framework for each of the squads and one of the, uh, pillars. of the prioritization framework are the KPIs that are aligned with the specific squad. So it just makes it easier and just more personal for the use of, you know, whatever it is that they’re working on. Um, and it means that, um, the alignment that we have with The squads is along with UX research.

It means that if we’re all kind of aligned with what the what test or what project we’re working on, it just makes things much easier. Um, so it’s like an extended squad almost. So you’ve got the squad, you’ve got the product owner, you’ve got the developer, you’ve got the business analyst, and then you’ve got kind of the experimentation specialists. You have a uX researcher, you have a designer who are.

And those three people sit within digital. So it just means that it’s, it’s almost like an extension, but those, um, kind of, uh, expertise are obviously sat within those teams. So, um, yeah, it’s working so far. It’s, it’s working well.

I think, like I said, I think the research aspect and the continuous discovery, point of view. That is something that we definitely need to hone in on more. Um, but it’s kind of like step by step when you talk about experimentation mindset. First thing is getting a test live and making sure that everybody is like really invested in it.

Is it exciting people? Are they motivated? Do people want to know the results of it? And then in the background, you’re trying to, you’re get prioritization framework in place and get people aligned with it.

Um, and then you’re trying to get people heavily invested in discovery projects. So, um, it’s kind of like a stage by stage thing.

Chris Gibbins: but it’s so good. But it’s very different from what you’re saying, because actually the ideas are coming from those different people from, from product. They’re not just coming from a centralized optimization team. They’re thinking optimization at the same time, aren’t they?

They’re thinking, how are we going to solve this problem, which we’ve discovered in research. And then you’re working together, which I think is absolutely wonderful. Key components of a, um, of a successful experimentation and well, product experimentation function in any organization rather than having the typical thing is when, when optimizations handled completely separately, you end up with clashes. You end up with a product team working on some initiative, building a feature.

And then there’s another team optimizing it, but sometimes one team doesn’t know what the other’s doing, and then you can have some clashes and misalignments, and I think it’s great to hear that you’re, you’re already thinking about those things and the future a bit more, even if you’re early on in your journey.

Annette Rowson: Yeah, I think we’re, we’re lucky in a sense where we’ve been brought into quarterly planning from a, from a product perspective in quite early. So you’ll see that when we go into those quarterly planning sessions, The client side tests are on the same boards as server side tests, so there’s, we definitely know that there’s not going to be any clashes. And also, even those client side tests, even though they might not be implemented by the squad, they’re still heavily invested because it’s moving the dial. for that team.

So those KPIs are still aligned with that squad. Um, so we are quite lucky with that in that sense. And I’d probably say we’re very lucky because it comes from top down. I think right from the beginning, I’ve had so much support from leadership that believe in the experimentation mindset.

So I’m very, very lucky. Um, I think one of the things that we’ve realized more recently, actually, is it’s not just the prioritization framework, it’s also a test hypothesis template. It just, because some people think, gosh, I’ve got to add like data inside, I don’t, I can’t remember where I got it from, and what’s the supporting evidence, and actually just giving them a template that they need to fill in just helps their memory and, you know, they can just attach whatever deck or take a screenshot of a dashboard or whatever. But it just, even if you’ve got somebody who’s not from a data background or not from an experimentation background, they, they don’t realize it, but they’re starting to think in that way because they’re writing it down on relatively easy form.

So,

Chris Gibbins: So, so everybody’s using the same hypothesis template now then, because that’s also an area that’s often done in many different ways in

Annette Rowson: yeah.

Chris Gibbins: of a large organization. So,

Annette Rowson: Yeah, I guess like, you know, we’ve had challenges when we’ve tried to implement it with some, Oh, I don’t have time to have time to fill it in. And it’s just part of the process, unfortunately, like, we have to know. And so I don’t know, we have to put your foot down a little bit sometimes just just to implement a process. But I completely understand on the other side, This is the newest function, you know, that’s, we’re, we’re asking everybody to add an additional step in their work day to day or whatever.

So, you know, I’m very conscious of that. So you have to be sympathetic also to be like, you know, I understand that you have a lot on your plate and this is like another thing that you need to do. Um, but this is, this is the way that we’re going to be doing it from, from now on. Otherwise it’s going to be very difficult to, to prioritize.

So if you’ve got something that is like, really going to move the dial on like one of our like, top KPIs and you don’t have any data supporting it, it’s most likely not going to get built. So, you know, the importance of making sure that they are brought into the hypothesis template is, is something that we’re working on at the moment. So,

Chris Gibbins: I know moving on to a different part of the opportunity discovery, like where do people’s ideas come from? How, how do you even uncover all the problems you want to solve in the first place? I know we had, I took this screen grab from something you used before, which was, um, explain what’s going on here in it.

Annette Rowson: I actually think that one of the most motivating ways that you can launch experimentation or an experimentation mindset within an organization is by doing an ideation workshop, um, and showing them interviews of real life people going through your websites, um, I didn’t actually, it’s the first time that I’ve actually seen en masse the reaction to an interviewee. Like, I, I didn’t realise that everyone was like, oh my god, that’s crazy. I think sometimes what happens, especially, Where we are at the moment, you know, we’re working so hard for Click and Collect to be successful and, um, making sure that the website’s, you know, up to date and like, there’s lots of things that, you know, is the forefront of our mind when we’re trying to drive and scale something, but actually, let’s just start with, So let’s just park all of that for a minute and why don’t we just figure out what our users actually think about our website.

And one of the, the first ideation workshop that we actually did, uh, was in partnership with Creative CX. This lady was one of the, um, interviewees and she just was so shocked that we even had a website and then was even more shocked that we had a click and collect offering. Um, and the joy on her face was just like next to nothing. It was, it was great.

And automatically just like that, you know, it was, we were utilizing at one of our quarterly, like big, um, digital and digitech meetings. like the reaction of the people in the room was like, Oh my gosh, like this, this is like one of our, a genuine user going through our website. Um, so this is one part, but it’s also the opportunities that were driven from this. I think everyone was quite shocked with how users actually navigate through the site and the reasons why.

So, We have things obviously like GA4 and ContentSquare that help us for quantification helps us with, um, journey analysis and funnels, etc. But there’s nobody there who’s like providing actual qualitative insights into the reasons why users do a specific journey. So it was really, really interesting. Um, and for us, within that, session.

We showed a series of, um, uh, of clips that were recorded. Um, and then we showed some quantification of a problem statement that was driven from that. And then we, we actually split the room into different groups to try and address that problem statement. And I think that that is one of the best ways you can get.

people on board. So if you think about it with Primark, like experimentation did not exist, so we’re asking people like merchandising, content, um, SEO, etc, etc. We’re asking teams who are not necessarily waking up and thinking about experimentation all the time, um, but they But you’re suddenly giving them a really fun activity to provide a solution to a relatively big problem because we’ve just shown them that this problem is quite big. And then you’re collaborating across the function or the department with people that you probably don’t work with day to day to come up with the solutions.

And I thought it was just such a lovely way to introduce the experimentation mindset and essentially. That’s all it is, really, is, is, you know, you’re using qualitative and quantitative data to come up with problem statements, um, and, um, ideate to create hypotheses. And from that, I think we generated about 35 different hypotheses just from that hour and a half. And I think it’s, I think the, I’d probably say it’s an underrated area, um, with other businesses.

I don’t think it’s, uh, utilized in a way because people don’t realize that it actually motivates people who are not from a data driven or experimentation background. Um, so if you’re struggling with that, I definitely would say that that is a, um, definitely a key area that I would hone in on if you’re struggling with getting hypotheses outside of your team to, uh, to, you know, come flying in. So yeah, definitely one way.

Chris Gibbins: And it’s interesting, earlier on you were talking about how your hypotheses templates, and I think you were hinting at maybe how things were done before in terms of just having a solution and running with it type of thing. And then sometimes, you know, oh, it needs to be backed up by data or it needs to have data supporting it. But actually, if you do things this way round, everything will have, be supported by data because effectively you’re, you’re looking for the data and the customer problems first, aren’t you? So therefore, naturally, all the ideas for 35 or whatever you said off the back of this are based on solving a problem.

So almost naturally, this, this improves your whole process.

Annette Rowson: Yeah, definitely. And I think, you know, there’s another layer of that happens, you know, which one’s more compatible for the website. Does our brand allow for that idea to happen? You know, does it, does it require.

An unreasonable amount of budget, like some super duper AI widget or whatever, you know, like there are things there’s some wild ideas that come from it. But I think, I think it’s so interesting. A, you start figuring out who your experimentation ambassadors are. And it means that you can kind of create a little mini network.

But also, I think it just helps people start thinking in a very different way. And so even though you know, I probably say that out of that 35, there are some that obviously, we can’t do. But I do think that It’s, it’s just a fun activity to do, first of all, and it makes sure that you are completely aligned with, um, best practices and it just means that the, the hypotheses that you are getting, whether or not you can do them or not, because of course, whatever, um, they, they do fit within the prioritization framework. So there are always going to be an ease.

There are always, always going to be some easy wins and things are definitely aligned and things that you can do that come out of it. Yeah,

Chris Gibbins: great. That’s really good to hear. And I think connected to this, I know we did an experimentation day or so, didn’t we? Uh, which was, which was a lot of fun.

And this was a bit different because this actually uh, Getting, uh, other brands and other people in a wider community. So, so we, this was a lot of fun. So this is effectively what you were doing internally, but on a bigger scale, where you invited many other people to help, um, uh, solve some customer problems and we did it in a fun way to, to almost promote UK, really, didn’t we? I remember, so there’s a nice picture at the top left of you doing the intro talk, um, and there’s one on the, on the bottom left of me, but, um, In the middle is we had all the teams doing the ideation once had been shown all the problems shown those really engaging usability testing videos and and also how they were quantified, like how many, how many people with effects and then on the right you can see presentations where all the different teams had to present their solutions to the audience and then we had gave out prizes and things.

so this, this was a lot of fun.

Annette Rowson: think the setup of this is really good because we are so entangled on our devices. This is one of the only workshops probably where you were just using. paper, scissors and glue. Um, and it just means that the interaction between your team members is very high and it means that that collaboration is there.

So I think for us, one of the things from a cultural perspective is that we are a growing, uh, digital function. And it just means that from a cultural perspective, we need that collaboration. We need people to be fostering relationships across, So, even from a cultural perspective, never mind experimentation, which obviously But let’s put that aside for a minute just from a cultural perspective. I think it’s a really good exercise to do.

Um, so yeah, I really enjoy it. I think the problem statement here was quite tricky because it was a click and collect versus non click and collect journey. Um, which, uh, was really interesting actually. And I think here.

The idea is because we invited other experimentation experts from other businesses, it’s just really interesting to see what hypotheses they come out with. And I think, you know, they do want to be challenged. So it’s actually quite an interesting, So, you know, for them to really, really have a think about it and, you know, get together with, you know, other experimentation experts with with other brands, um, and and create a prototype. Um, I mean, they look like they were having lots of fun.

Chris Gibbins: Yeah, don’t know. And, uh, I mean, I always love having scissors, and, um, it’s a bit like Jupiter, isn’t it, for those who remember that, um, session. So, no, no, that’s good. I wonder, I think one of the, one of the key things with the session you ran internally as well was getting people to think beyond their initial idea.

I know that that’s often a problem in product teams with designers sometimes as well. People they fall in love with one idea, they get all behind it, and then it’s, it’s, it’s sometimes, it makes everything after that, like, it makes many aspects of experimentation of, you know, How, how can my idea not win? What happens if it loses? And, and all those challenges that are quite difficult for people when they start off.

Getting people to think beyond their first idea and have multiple solutions to a single problem, I think is a really good kind of educational aspect of, of just getting people ready for experimentation.

Annette Rowson: I think on that. I mean, I’m also a victim of this sometimes, and I have to kind of check in all the time to be like, is it the right thing to do? I do think collaboration obviously helps so much here, and seeing the different hypotheses that come in. But there is a step in the workshop that we did, and the workshop internally as well, where we did something called Crazy Eights.

So everybody has a piece of paper, you fold it. So you have eight boxes in the end. And I can’t remember. Is it, you have a minute per,

Chris Gibbins: Eight minutes for eight ideas,

Annette Rowson: eight minutes for eight ideas. So you try on average, trying to hit a minute per, per box. And what’s quite interesting is what your brain does. Because your brain just goes, Oh my God, I need to like pull my ideas down.

And that’s why you kind of say, um, you’re like a gamble one. So that’s why some people put like AI, crazy, crazy widget.

Chris Gibbins: It’s all about the wildcard idea, having one of

Annette Rowson: That’s it. Um, so that’s why you do see those, but it’s interesting what people put in those. Um, and then once you have those squares filled in and then you speak to your teammates about it, what usually happens is that you can actually merge a bunch of ideas into one. So actually your hypothesis becomes a lot more valuable all of a sudden, um, and your, and your idea.

So, um, I do think that’s a very good process. Definitely to do. Um, and our UX research team, who are also relatively new alongside experimentation, are very much so brought into that. So we are, we do kind of, we do move as one, uh, when it comes to that.

So, uh,

Chris Gibbins: Sounds great. Um, actually moving on to a slightly different question. You mentioned earlier about, um, the, a bit of a top down approach in terms of buy in for experimentation, because I realized it’s often a struggle for teams out there, for people out there, they might be doing really great work in their little team and But it’s hard to, and you know that this could benefit so many other teams out there. If only I could kind of spread this culture of experimentation.

We know it’s going to benefit so many other teams, but it’s quite tricky. when you’re in, when you’re effectively busy doing all the work to change an organization in that way. So how did, how did that come up? Cause it sounds like at Primark, they are fairly bought in at a senior level now, uh, which has made a big difference.

Annette Rowson: I mean, yes. Is it just me? Am I, are they just brought into me? No, I’m joking, they’re not.

Um, they, um, I’m very, very lucky here. And I think I’ve said it so many times, so I might as well say it again, that I was a real nomad before I came to Primark. I’ve kind of like floated and, you know, kept trying different things and went from kind of roll to roll, trying to, trying to find my, my home. And I think Primark, because of how they, value experimentation and, um, data analytics.

Um, it’s, it’s definitely something that is of immense value to me. Um, and that is mostly it does become it’s because it comes from top down. Um, I think one of the reasons is the ROI piece. I think it’s incredibly hard to get ROI.

when you’re optimizing a site. I think, um, I think when I started doing the program or started kind of implementing the first steps of, of, of the program and trying to get people on board, a lot of the things that I was hearing was, Oh, we can just do pre post analysis. And I was like, no, that’s, I get it, but no, um, and you’re never really going to know the effect of a change if you don’t do a traditional good old A B test. Um, And it just means as well that when you, you have to understand what makes your stakeholders tick.

So like, for example, if you’re the team who’s in charge of, um, user generated content, let’s say if you’re that team and you’re like, right, we’re going to spend some money on an agency that’s going to, you know, help us get this on board. And we’re going to do this and we’re going to ask for loads of dev work to get that implemented, but you don’t know the ROI on it. Like at all. Like, so what about if you do do an A B test?

And yes, unfortunately, you do have to wait a tiny bit to get those results. But isn’t it more fantastic if you go back to the business and say, Hey, that thing that, you know, my team’s been working on for absolutely ages. And that thing that’s going to really move the dial. I figured out through an ABE test that it’s brought us X amount of money, and I think that’s an amazing piece.

And once you bring stakeholders on that journey, or even leadership on that journey, they start getting on board. It’s like everybody Well, I’d like to think that most people that work within an e commerce or digital background, you, you do have an element of a commercial mind, right? And the ROI piece is, is, um, is huge. So, um, we do track that.

Um, and it’s something that I am championing from a, uh, experimentation perspective is that, you know, we do need to go back to the business and report the ROI. not just for us, but for all the other teams that have contributed, collaborated and, you know, supported us on our journey.

Chris Gibbins: So that’s been one of the most successful ways to get the buy in. To actually talk to different teams, find out, I mean, what are they trying to drive anyway in terms of their value and then sell them how the idea of A B testing as a way to really figure out what the ROI is, figure out what impact they’re making as a team.

Annette Rowson: Yeah, definitely. And I think, you know, this is just a challenge that we’re not a challenge or an opportunity that we’ve had for digital, where we’re bringing people on that journey. We’re like, you know, we can, we can show you how much impact that, that thing that that idea that you were doing has, has created. Um, Um, but what about if we went to the rest of Primark and we went to, for example, a men’s buyer who was like, Oh, I’m not sure if, you know, this lifestyle image is going to work more than this kind of flat lay white background image.

Um, and we were investing lots of money into, you know, taking those. Um, those shoots. Um, but we’re not sure. It’s like we’re, you know, even outside digital and getting, you know, our teams who are traditionally from the bricks and mortar.

How do we then bring them on that journey? You really do have to pick on the things that make them tick and the big projects that they’re working on. Um, not that I’ve been successful in that quite yet, because obviously we’ve just started this, but it is something that, you know, we do see as a huge opportunity. Um, and um, that it just brings in that collaboration piece again, you know, making sure that we, you know, the right people in the room and the right ideas are coming through and people are brought into kind of the process.

That ROI piece, but it’s definitely, I mean, people don’t like a process, but they definitely like a bit of ROI. So

Chris Gibbins: exactly. And it’s not just the uplift that you can achieve in a particular area. It’s the cost savings as well. I mean, I think the, um, especially in retail and e commerce, you need to think about this, the costs of doing a different style of shoot, for example, the costs of having all model shots versus cutouts.

And like you say, maybe lifestyle versus non lifestyle is

Annette Rowson: also like the number of images that you have in a carousel, like, do you need a zoomed in? Do you need a full body? Do you need, you know, do you need the lifestyle and the flat lay? Do you need a zoomed in zoom out?

Like, you know, I’m not saying that it doesn’t drive. saying that for your experiments will tell you what’s right for your business. You know, it’s not, it’s not one fits all, you know, the experiments will, will guide you and in what works for the business. Um,

Chris Gibbins: totally. And also when it comes to, we’ve had success of just, I mean, it’s not always about looking for an uplifting conversion rate. I mean, many of our clients, we test third party tools. So when a tool, when a tool provider comes in and promises a massive uplifting conversion rate, no worries, sign here.

You’re going to definitely get this uplift from our special AI tool, for example. What a lot of the work we do is help our clients to build an A B test to really, truly identify whether it adds value or not. So you can, you can test so much, so much more than just the UI on a website.

Annette Rowson: yeah, definitely. I think. I do think mitigating the risk, also things like, um, you know, when you implement something, sometimes you implement it incorrectly and it fires errors and all this sort of stuff, like it’s much better to approach that with an experimentation mindset. Um, there is another thing that I was going to say, and it was around, sometimes it doesn’t win or lose, but You know, maybe.

Chris Gibbins: would that, how do you get people to, to, um, come on board about idea that, that they’re fantastic or all the knowledge they have over the years isn’t necessarily going to result in a winner every single time.

Annette Rowson: Exactly. And it’s, it’s quite interesting because I think because the, the program was slightly delayed, we had some very passionate stakeholders with some interesting ideas that they wanted to put live. And once they were put live, you know, they were really shocked by the impact, whether one lost or, you know, didn’t do either, just nothing happened. Um, I think it’s, it, I do find it quite an interesting topic is, and I’ve, I think I’ve said that before, um, gosh, I talk so much, um, where I talked about, you know, a big part of experimentation is stakeholder management.

It’s a huge part of what we do. Um, and it’s that fine balance between, you know, managing stakeholders and your results, because. It’s the truth. Um, and also making sure that you’re keeping the motivation going if they get the response that they don’t particularly want to hear about a test.

Um, it’s, it’s definitely a soft skill that is needed. Um, somebody in the team, doesn’t have to be everyone, um, appreciate that everybody has their own kind of buckets of expertise, but it’s definitely something that should exist within the team.

Chris Gibbins: No, no, very cool. Um, actually, I thought one thing I would share, because I know, uh, you talked about, um, pre, pre and post testing, you mentioned where people would launch something and then try to detect afterwards, whether it worked or not. And of course, we all know, but try detecting a five or even a 10 percent uplift, um, week by week. Actually, it’s a nightmare.

So one, one task, because we had another client who was suffering from this a bit. Um, so we actually did an AA test when nothing had changed whatsoever over a few weeks. on a website. They’re very early on in maturity level.

And the team then presented, um, the results of this AA test that ran for two weeks. Um, sorry, just after the one week. And, um, we used some of the data to show them the variant, to show them the things, but data is very different week after week. So the first thing we did was to When there had been no changes, we just presented them with one week versus another.

And naturally, there was a 5 or 10 percent improvement because the weeks just change all the time. So that solved the first problem. The second problem was that actually they like to call a test too early. That was it.

And the solution, which is another problem, you know, when people start looking at results after a few days of an A B test. Um, so actually that’s where we use the A A test. So we actually took an AA test and then we, um, we demonstrated that after three days, even an AA test, when there’s no difference at all, you can see, you can see results, dramatic results. And it was, it was almost like, uh, I know we were playing with, uh, or they were playing with their stakeholders a little bit, but those, those tactics can be quite useful to educate actually, to show them differences and things.

Annette Rowson: yeah, it’s hard. It’s quite hard to explain unless you have evidence. So I do think that’s a, that is a good approach. Definitely quite lucky.

We’ve definitely shifted from the pre and post discussions these days. So that in itself has been.

Chris Gibbins: It’s a job well done, Annette.

Annette Rowson: Thank you.

Chris Gibbins: So, I wanted to end on the last question, which was it sounds like things are going great, very early on, um, your, you just even have run some server side tests this early is very impressive, to be honest. Um, but I think that comes down to your approach of including teams in everything you’re doing, rather than just taking it and holding it to yourself. Um, And I think that’s a, that’s a, um, a testament to that. But what are the biggest challenges you are currently facing and, and how are you addressing them?

Annette Rowson: I’d probably say that. On a bad day, I can be relatively impatient. I think that’s one of the things I don’t think patience comes to me naturally. And I think if people were to be honest, they probably Um, and I think starting a program, you do have to have a level of patience and you do have to have a level of compassion and understanding of stakeholders.

So that’s definitely something that I personally have overcome. And I think that’s probably one of the reasons why. So far, the program has been quite successful. Um, I think where are we going next?

I think it’s definitely about that scaling up and, you know, the speed to set up tests is definitely an area that I think we need to work on. Do we have the right, um, process in place for test post test analysis? Um, Are we doing the right? So there’s one thing around prioritization framework and make sure we’re choosing the right test in terms of impact.

But are we also choosing tests that are bringing in other stakeholders into our little experimentation world? Is there more that we can be doing in terms of the types of tests that we are building and maybe prioritizing tests? still within the prioritization framework, but also from a cultural perspective, are we kind of like ticking a little

Chris Gibbins: That’s really. That, that’s super interesting because if you were purely doing it, because your aim isn’t just about driving the ROI straight away, because otherwise then you end up with, you’re probably going to focus on just a couple of your core teams. But, but it’s because you have another ambition, isn’t it, which is to scale experimentation to all the different teams at Primark. So you have to consider these other, you have to, that’s really interesting, you have to consider experiments just to bring in different, different team members.

Your

Annette Rowson: know, when I started this, I was like, no, it’s what the framework is saying, and then you start realizing after a while, sometimes you have to think from a cultural perspective and also think about longevity of the program and, you know, How influential certain people can be and, you know, what are the things that they’re driving, um, and kind of connecting, you know, the dots. And so the culture, a culture, a culture column within the framework, I wouldn’t do that, but, you know, almost like a, an invisible column, um, should sort of be there because you’re thinking about, because you’re starting the program, does it involve a lot of people that are going to be excited by it?

Yeah, and then after that it’s around communication. So, you know, we’ve been working so hard on making sure the tests are going live and we’re getting people on board and implementing the framework and getting used to a new testing tool, etc.

Chris Gibbins: That’s right, no, no, I think one of the ones we mentioned before was this, people very used to a long list of JDIs, so just delivering. Got so used to this way of working. If I persist, keep on asking it, it’ll happen though. So I think it’s, it’s how the product teams are very much about that’s what they were used to.

That was the world that we’re used to.

Annette Rowson: Yeah. So I guess one of the other challenges we have that still kind of exists is that kind of just do it mindset and not implementing the experimentation mindset is that we do have that support top from the top. Um, and For us, there are pockets of people and teams that are on board with the experimentation mindset. But I also think that people accidentally sometimes just go into a just do it.

Um, and also on top of that, it depends on how desperately they want it. So sometimes people take a little back, back alley route to try and get a just do it done. And then it comes full circle. Because we are aligned with product, we’ll find out.

And then I go, hang on, how did that get on the road map? Uh, let’s go back a bit. Uh, I think that needs to be a test. Um, and again, because I have the support, uh, from, from leadership, usually, um, it does become, uh, an experiment because the impact, um, is great, but also the risk is great.

So if it’s something that’s a, an incredibly huge change, like it needs to be an experiment because we need to mitigate the risk.

Chris Gibbins: So explain the just to it in a bit more. Let’s tell me a little bit more about that. So is it because they, they are still thinking in this kind of delivery roadmap type of frame of thinking where they, They just create a road map of features that they and functionality They know they want to deliver because they got used to working in that way And they is that what they fall back to that that way of we just need to almost Not imagining that it might not work. Is that right?

Or

Annette Rowson: Maybe. I, I don’t, I sure don’t think so. I think it’s more, we have stakeholders that have been waiting for things. to happen for a couple of years.

Some people have been waiting for a long time and I completely get it if you’ve got a big team and you’re the thing that you’ve wanted to get changed has been de prioritized and de prioritized because we you know we’ve been scaling so rapidly and you know there’s been so much that we’ve needed to do on the website. Naturally there are going to be stakeholders that that feel like the thing that they are trying to focus on has been deprioritized for good reason. But still, nobody likes to be deprioritized. I certainly don’t like to be prioritized.

So, you know, I understand where they’re coming from. And sometimes what happens is Because they’re so eager to get that feature or that, you know, that component or whatever builds, you know, they’re like, please just do it. Just do it right now. Like, let’s not test it.

But then you sort of have to remind them of the ROI. Like, okay, you’ve been waiting for a long time. What is a couple of weeks? Like, Just wait, wait, let’s just try and get that ROI.

And also it’s just fostering that relationship. You know, I guess before there weren’t go to experimentation individuals in the business, whereas, you know, we are around going, you know, we’re on your side where we’re here to help you, but you need to go through this route because you won’t get that ROI.

Chris Gibbins: hmm. No, no, absolutely And then just moving forward to suppose when you get a bigger team, you can have that team supporting More the different product teams can’t use effectively. They can all work in parallel when it comes to experimentation. So I imagine there will be, it will be less of a, um, there won’t be blockers because there’ll be continuously experimenting within their, their teams within their squads.

Annette Rowson: Yeah. And also I think because there’s, you know, we’re going to be working on the alignment with digital analytics as well. It just means that the alignment of the digital analytics project will also be aligned with experimentation. I mean, analytics is, you know, it’s not just experimentation, but there will be a proportion of it that is dedicated to discovery.

Um, whereas everybody, because of how quickly we’ve been doing things, um, you know, we’ve definitely been, um, Kind of. trading focus, but also, you know, the process of, for example, like tagging and making sure that the, you know, everything’s being tracked, right. That’s been like a really big focus for us. Whereas now it’s like, again, as much as we’re also trying to install experimentation mindset and optimizing the site, you know, we also need to make sure that that resource is available.

So there are people who are dedicated to discovery, um, and, and doing those deep dives. So it’s definitely, I always say that, you know, uh, UX research, design, digital analytics, and experimentation, we really should be moving sort of as one, like there should be an alignment there. Um, and we should be speaking the same language as, as products, because if there’s a misalignment that, you know. There’s going to be a miscommunication or, you know, there’s going to, people are not going to be brought on, you know, whatever tests, um, you know, we want to get built, like the, the alignment pieces, um, it takes a while to get right, but it’s definitely worth it.

Chris Gibbins: Cool. Fantastic. And so what’s next for Primark’s experimentation program? What does the next six month, one year look like?

Annette Rowson: Um, I think we need to scale. Well, we need to scale our service side test and our, um, client side test. I think when we started the program, it was kind of like quick wins. What can we do really quickly?

And what’s not going to because we’re kind of like from a we’re from a risk adverse. And, you know, we don’t also want, we didn’t want to start building very complex tests that was going to rock the boat. So we had to do very simple tests at the beginning, um, and, and those quick wins. Um, not saying that quick wins and easy to implement tests aren’t impactful.

Um, but I’m excited to start building some, some, you know, slightly more complex tasks. Um, things that are, um, going to be aligned with, um, teams that we, that are untapped from an experimentation point of view as well. Um, so I think that’s what definitely one area that we, um, focusing on. The other piece will be segmentation.

So definitely in the next six months we’ll be looking at segmentation. What’s our most valuable segments that we should be using? Um, what experiences or what components do those segments want to be seen versus others? Um, and enhancing that experience for our users from a segmentation perspective.

And then hopefully I always say light years ahead, probably not light years ahead, but actually start doing a bit personalization. So, um, Definitely working on the plan for, um, segmentation and personalization going forward.

Chris Gibbins: That’s cool. Because your users have very different needs, don’t they? I mean, just thinking about the journey from Primark at the moment, I might well be just researching a particular item just to, just to see if it’s in stock in my local store. Or I might be wanting to buy it now and click and collect or there may be maybe a hundred other reasons Each of those I might have completely different user needs and and motivations to the next user to the next customer So that’s where the opportunity lies isn’t doesn’t it for personalization?

Annette Rowson: Yeah. I think even going back to the beginning, I think our journey is so different from others. You might not even know about click and collect at Primark. So it’s like, what do we do for that segment of, of, of users, you know?

So, um, I think it’s, It’s complex in a way where we have to learn from our own data and insights. Um, uh, but then on the other side, it’s really exciting. Um, and, uh, we will probably always be surprised. Um, but no, I’m, I’m very excited.

Chris Gibbins: mean, now just in my, my instance, my logged in state in Primark has my store saved next to it, doesn’t it? My local store. So even that, of course, is a time saver for me. And that’s something you know about people now, don’t you?

For those who are logged in versus someone completely brand new to Primark. So yeah, that sounds, sounds like a good six or 12 months ahead for you. Um,

Annette Rowson: I’m very different from the previous six, six months. Um, But yes, I’m, I always say that I always think here at Primark, I always thought, um, Oh, like, well, this is this. These are the things that we’ll do. And then you come out with something very, very different after the next six months, because there is so much to do.

But also we we live and work in a forever changing environment. So you have to pivot, you have to adapt. And it just means also that the impact and the priority changes all the time, too. So, um, yeah, really exciting.

Uh,

Chris Gibbins: And what would you, um, I mean, from your perspective, because a lot of people in your, in your shoes as such, who are, who are building up experimentation programs, getting all of this going, what, uh, tips or recommendations would you, would you want to pass on to other people, would you say?

Annette Rowson: if you are kind of leading and creating an experimentation program and what it depends really, if you’re sort of on your own, which I was, and it was the first time where I’ve been a solo act in, in a, in a long, long time, I think I came from a place where I had like quite big teams around me. Um, So it was definitely a big cultural shock. I think the first thing is actually the experimentation mindset and looking at your tool stack. That’s one of the biggest things, you know, you have to know that what the capabilities that you have and, um, also bringing people on that journey, whether that’s, whether you kind of brand it as experimentation mindset, sometimes that’s, that can scare people at first, but definitely, definitely like a data driven culture.

First, I mean, that’s the first thing that we did here. First thing we did here is that we realized that, um, there was only, uh, data and a data digital analytics that we’re using, you know, our, um, analytics tools. So, you know, now we have a weekly, um, data and insight surgery where it’s an open door where any stakeholder can come in. All their teams can come in and ask how they set up a simple dashboard or, um, how do they find, um, I don’t know, like, how do they set up an alert in content square or how do they, um, analyze a journey for a specific user segment?

Like, these are the sort of questions that we get in surgery every single week. Um, but that open door policy, I think, is really important. really important because typically, especially in other businesses that I’ve worked in, analytics is kind of like this wizardry that happens, you know, behind closed doors and nobody really knows what, what happens. That you submit a request and something gets spat out and it looks really intelligent and you’re glad you have it, but you’re not sure how it was created.

We’re sort of trying to change that mindset here and say to people, you know, You can, you know, do this and, you know, you, you, you, you can actually grow a data driven mindset. Um, so that’s kind of one of the initiatives. I would definitely say advise somebody. And if you are worried with that, you know, we also have data champions.

So we have a smaller network of individuals that are placed within different teams, um, even the non data driven teams that we upskill. So, you know, the more you hone into a smaller network of people that belong in different teams, the more you’ll grow your and foster that relationship. So when the time comes, when you’re like, Oh, we’re starting this thing called experimentation, you’ve already fostered that relationship and you’re, you’re ready to kind of tap into, you know, the right people and you know who you can speak to and you know, who’s going to be an ambassador. Um, so I definitely would, um, advise that too.

Um, the other thing is the alignment of prioritization framework. I really could not emphasize that more. It’s incredibly important. I think, um, here I can, there was a couple of weeks, I think I said a prioritization about a million times.

Um, but it’s just the culture that we come from. You know, we come from this kind of like super speedy, super efficient, just do it. Like things need to just go. Go, go, go, go, go, and I’m coming going, Okay, hold on, everyone.

Let’s just think about this. Let’s just prioritize. Um, and I think once you get that And the alignment is that people are bought into it. And the only way you can actually do it is by carry on showing it to people like, Oh, you have an idea.

Let me put it into the new kind of like, they’re watching you type it into your prioritization framework. They’re watching you kind of score it. They get bought into this idea and then, Oh, where is it on the framework? And they get bought into this kind of culture.

Um, and, and also it means that the transparency is there. And actually the most important thing is I’m not everyone’s worst enemy because then suddenly the framework is this kind of non bias thing that, you know, tells everybody what the answer is and it’s not me just going, I’ve decided, you know, it’s like, it’s the framework. So,

Chris Gibbins: Cool. Well, thank you so much, Annette. Thanks for sharing all those insights and your whole journey on Primark so far. It’s been fascinating to hear, and it sounds like you’re on an upward trajectory, which is fantastic to hear.

Um, one last question. Okay, and you’ve got a choice. You can either share a book you’re reading, or share some podcasts that you like to listen to. What would you like to share with the audience?

What’s, what’s, what’s top of mind at the moment?

Annette Rowson: I, I’m one of these people I think because my Brain is kind of like in so many different areas at one time. I think with podcasts, I try and like, listen to things that are relatively easy unless I’m, you know, on a long drive or something, I’m like, right, I’m ready to listen to something. But, um, I just, I tend to, I like listening to people’s stories. I think it’s, I think people’s life stories or, you know, the way people have dealt with things.

Whether that’s with work or personally, I think, you know, sometimes you can take real inspiration from that and, and try and apply it yourself. Um, so one of them is How to Fail by Elizabeth Day. Really easy to listen to. She invites really inspirational people to come on her podcast and they talk about challenging times in their life and how they, they’ve overcome it and how to combat it.

Um, and, uh, It’s just a really easy listen. So I tend to just listen to that when I’m cooking. Um, so if anybody fancies it, I would suggest, uh, listening to that. But it’s pretty much, uh, a couple of people babbling along about their personal lives.

Um, but, um, but also their careers and there’s some, honestly, some real inspirational people there. So sometimes I will come, I’ll come away from like listening to a podcast. I’m like, gosh, like, I can’t believe that they overcame that and it just kind of makes you feel lighter. Makes me feel lighter anyway.

Chris Gibbins: great to hear. Um, I suppose from my point of view, it’s only fair I, I give a couple isn’t it, of podcasts, I think. And my, I think my, my favorite time to listen is when I’m cooking also, but also when I’m, when I’m running, I find running quite, quite a good time to listen to podcasts, I think. Um, Current ones are Lenny’s podcast from a more product side of things And i’m also a bit of a Freakonomics fan as well.

So that podcast is amazing So those are my two favorite I think at the moment, but um, no, I think we’ll wrap up there Thank you so much again

Annette Rowson: Thank

Chris Gibbins: and see you everybody

Annette Rowson: Bye.