Experimentation in the Hospitality Sector

[00:00:00] Mareen Thomas Cherian: Hello, everyone. Welcome to ConvEx ’23, VWO’s annual virtual summit for growth and experimentation professionals. 

[00:00:16] My name is Mareen, and I’m a group manager of content at VWO. Thousands of brands across the globe use VWO as their experimentation platform to run A/B tests on their websites, apps, and products.

[00:00:30] I’m excited to have Alexandre with me here, who is the Head of Experimentation at Accor Group of Hotels. 

[00:00:38] Hey, Alexandre, excited to speak with you. 

[00:00:41] Alexandre Suon Perhirin: Hi, Mareen. Thank you for having me. Really excited to be part of this amazing event. 

[00:00:48] Mareen Thomas Cherian: So, Alexandre, let’s get to know you first. 

[00:00:51] Could you please tell us about yourself and your journey as an experimentation professional?

[00:00:57] Alexandre Suon Perhirin: Sure. So my journey started about 10 years ago. I studied computer science in university and then I focused a bit more on finance for my Master’s. I was supposed to start basically being a frontend/ backend developer, so full stack there back in the days for an internship.

[00:01:18] And the first day they asked me to actually build on an A/B testing tool, an experiment. So that was the first task that they asked me, and I really loved the concept. You know, I think 10 years ago, that wasn’t something that you would learn at university, right? You wouldn’t do A/B testing at university, you wouldn’t learn about experimentation, it was something still fairly new, at least in Europe.

[00:01:39] And the first time I created an experiment, I wanted to do that thing for the rest of my career. I thought it was an amazing way. I always really liked the strategy side of things in digital. The one of the course that I was doing was digital strategy, but it wasn’t really talking about a strategy, right?

[00:01:57] And learning about how a business in general could make better decisions. Even as an intern, I thought it was fantastic. And when I learned that this wasn’t something that everybody wanted to do, I basically decided to do that as a career. So this is really how I got started. 

[00:02:14] I started a small agency in London. And then for the past 10 years, I’ve travelled a bit. And which led me to joining Accor.

[00:02:25] Mareen Thomas Cherian: That’s awesome. It’s like experimentation has become your passion. 

[00:02:30] Alexandre Suon Perhirin: Yeah. 

[00:02:31] Mareen Thomas Cherian: How did you start with it? And why did you choose CRO as a field? And how has your experience been overall? I mean, if you look at past 10 years from there to now, how has CRO as a field evolved, and what has your experience been?

[00:02:48] Alexandre Suon Perhirin: I mean the field has evolved tremendously. And during my career has evolved as well a lot. At the beginning it was really about building tests, right? Again, 10 years ago, I was in a British agency which is helping mostly French companies. And when we would come in, most people wouldn’t even know what was an A/B test.

[00:03:07] We would have to teach to people what was an A/B test and then we would start building kind of random tests for them and that was the business model. Yeah, I remember using VWO as one of my first tools. I think back in the days there was really only VWO and Optimizely. And now VWO is still a growing company and Optimizely is a bit less present, let’s say. And so this is how I got started. Then I realized I need to have a lot more skills than just being able to build tests you know what I was realizing is that many many companies didn’t have a full strategy on building better products.

[00:03:45] So I decided to first get better at data. So I went to Gibraltar and I became a CRO manager. But at the time I was leading also the data part of things. So I was a data analyst at the same time. We started implementing tag managers. We started also doing lots of product analysis and led everything to personalization.

[00:04:07] Again, that was in 2014. 

[00:04:10] 2015, so it is way less common than what is nowadays leading to tracking related to eCommerce, etc. So at the time, we were really hands-on. And once I got a good grasp on analytics, that’s when I wanted to get a bit more into testing, and I joined at the time at Optimizely as an engineer. And that experience really helped me understand all the different functionalities and technicalities about experimentation. 

[00:04:40] So, I was really blessed at being able to first being a developer, then a consultant, then an engineer. And then later on I joined the creation of an agency in the UK that still exists called ‘Creative CX’.

[00:04:53] And then did my part over there and decided to go back to France because the industry is so different. I felt like it’s an entire new world of France. It’s not as mature as some English speaking country. So it was really interesting to come to the market and I had the pleasure to join Accor four years ago. And we went from zero tests to a good amount of tests now.

[00:05:20] And, now not only experimentation is about testing, but it’s also about thinking about the strategy, which is really, really, really important, actually! Which is the whole game in the end. 

[00:05:31] Mareen Thomas Cherian: Do you still feel the need to make the people aware of what is A/B testing and how important it is or to educate the value of experimentation?

[00:05:41] Do you still feel the need to do that? 

[00:05:44] Alexandre Suon Perhirin: Yeah, of course. I mean, I think if I look at Accor for example, everybody has heard of what is an A/B test and everybody nowadays have actually been in contact with some kind of testing in the previous experience, I think it’s very common. Don’t remember who was telling me this, but they were saying an A/B testing tool now is a commodity.

[00:06:03] If you look at everybody around, everybody has tried some form of A/B test. Whether through you guys or even through Google Optimize back at the time, because it was a free tool. So it really helped, I think, lots of companies to get started with A/B testing. However, lots of them haven’t done what I would call experimentation, right?

[00:06:23] That whole process of finding the right hypothesis, the right prioritization, solving problems, and that’s why I’m head of experimentation and not head of A/B test. 

[00:06:34] We constantly need to educate people on how to do better this practice. And that’s mainly what I do. Even a couple of weeks ago, I was talking at another conference called the product conference in France, and the chance to speak with a dozen of product managers and lots of them were saying, well, we do lots of tests, but we don’t really know actually why are we testing? Or what kind of impact it actually has on our product roadmaps? And it was really interesting to give them maybe a few tips on how they could do a better job. 

[00:07:08] Mareen Thomas Cherian: Hospitality, as everybody knows, is a multilingual and multicultural industry.

[00:07:14] So how do you handle testing for different languages and cultures? 

[00:07:19] Alexandre Suon Perhirin: No, it’s interesting. Well, if you look at first the industry overall, I think the hospitality industry is one of the most complex, competitive industries, right? If you just look at the landscape of companies, I mean, whether you’re in the states, you have the Airbnbs that are now across all the regions in the world.

[00:07:35] In Europe, we have Booking.com. In the US you have Expedia. So you have lots of what we call pure players, like extremely, extremely good at digital user experience. Then you have also old groups. Like Accor that used to own hotels and have very, very powerful brands. So you have Accor, you have Marriott, you have so many, so many large groups that owns sometimes 30, 40, 50 brands.

[00:08:02] And then you have those new-comers, like I said before, a bit like the Airbnb. You know, doing a bit of mix and that is super, super good as well, the user experience. So I’m saying this because first of all, it’s very important to organize that digital experience is key for those industries and if you think about the overall user journey, most of us now, when we go on holidays, we start by searching on the web. 

[00:08:27] So everything is about being digital. So once you said that, you realize that indeed, according to either languages or your where you are based in the world, while people don’t have the same usage or the same desire in terms of user experience and to give you a concrete example, Accor Digital experience, the landscape in China was so different that we decided to create a different product.

[00:08:55] So, for example, myself, I’m optimizing for the entire world, except China. China has its own team that takes care of the process of thinking of optimizing the page. Having said that, also, if you think of a company like Booking.com, they’ve actually acquired multiple companies because they realized that they couldn’t provide the best experience within a single tool.

[00:09:19] So you have Booking.com that is extremely strong in Europe. In Asia, I believe you guys are using Agoda. I remember when I was in India in January, everybody was sending me links to Agoda. I was like, I’d never heard about those guys before. So it was really interesting and the experience is quite different and funny enough. Sometimes you write an A/B test that were like losers for us and on Agoda, that’s what they did. So I was like, okay, that’s quite different. If you think about it as well, Expedia has come for some part of the world. And I don’t remember the name, but Booking.com has also another company for Australia.

[00:09:55] So first of all to answer your questions, it’s very important. Yes, as soon as you have a good amount of traffic from different parts of the world to experiment or to improve those products. And the way we’re currently doing it we’re not doing it perfectly, to be honest, we’re doing baby steps.

[00:10:13] So if you think about this form of customization two or three years ago, and now we’re starting to realize that there are different patterns. So we’re starting to present differently, for example, for Americas, because the way they are visualizing price is very different from Europe.

[00:10:33] And I know Asia, it’s our next target. But what’s really hard is when you’re not living in this part of the world. Well, to learn and to understand what you’re supposed to test different. So we are still trying to figure out how we’re going to do things. Usually what we do is at least we split our test results per region of the world and we see what’s the winner? What’s the loser? And it’s funny because quite often we actually do see a difference. 

[00:11:04] Mareen Thomas Cherian: Definitely have Agoda, but then there are many other players as well. So yeah, competition is rampant in this industry right now. 

[00:11:12] So how does Accor Group use experimentation to improve its customer experience across its different hotel brands?

[00:11:23] Alexandre Suon Perhirin: I could speak about that for four hours and hours. You know, it’s a very long story. So first of all, we haven’t been at this, would say, senior level for a very long time, right? We started our experimentation journey about four years ago. So again, the group is over 40 years old. Accor existed when the internet wasn’t a thing yet.

[00:11:47] So they are one of those players that took a bit more time to realize that digital is really important. And once they realized it’s important, now they’re putting lots of efforts and energy and money into it. So, four years ago, they launched a new site called [] and on this new site, the objective was like I believe, to provide our users the best experience possible.

[00:12:13] So that’s when the journey really started. So, when we started using experimentation, it was called A/B test, and we’re just running a couple of tests per year to validate a bit random hypothesis. To be totally transparent, I think, like many other companies they heard of the word A/B test when you started.

[00:12:34] So they were doing A/B test. Again, there wasn’t any strategy behind it. And it took us 2-3 years to start realizing, well, if we just launch some random tests here and there, there’s not going to be any impact in the end, so we started to increase our velocity there and also our presence as a team.

[00:12:54] So we created an experimentation team where we are nine people at the moment that has multiple objectives. 

[00:13:03] Our first objective is definitely to actually deliver A/B tests. At the end of the day, we want to be testing as many things as possible, as long as it’s relevant to test them. So that’s our first objective.

[00:13:15] Our second objective is obviously to find opportunities to improve the user experience. So not only focusing on building a test build, looking for things that we’re supposed to test, and for that we have many tools. 

[00:13:28] So we have things like obviously session replay, which is very useful to see if a customer or user is able to use a component the way it’s supposed to use it. But we have also obviously many product analytics tools, we have ContentSquare, we have GA, so all of that. 

[00:13:49] Our global objective is also to look at the vision of the product and see how we can help them better reach that vision. So that’s really how in a few words, Accor is using experimentation at the moment.

[00:14:01] If we go to something a bit more day to day, experimentation has to, again, build the A/B test, report on the results, train all the new people that are joining the group constantly. It’s a very large group. On the digital team, I think it’s 2,500 people. So you have all these people coming and going. You constantly have transformation, so you have to adapt.

[00:14:24] The positioning of your team as well to those things. So it’s a form of center of excellence plus product strategy team. If that makes sense. 

[00:14:35] Mareen Thomas Cherian: So at Accor, experimentation is a part of history. I saw it in the history section also and you have experimented with hotel concepts that allow travelers to design their own homes and rooms. So how does the Accor Group leverage experimentation to develop and launch new hotel concepts and brands? 

[00:14:55] Alexandre Suon Perhirin: So, I wouldn’t be able to say on this part, I’m only really working on the digital experimentation.

[00:15:02] So I know there’s lots of experimentation and innovation that is done in the group, but that’s definitely not something that I’m familiar with. 

[00:15:10] Mareen Thomas Cherian: What are the most successful experiments that the Accor group has conducted in recent years to optimize its revenue management strategies?

[00:15:22] Alexandre Suon Perhirin: That’s a really good question. I can’t share any numbers, obviously, precise numbers. But I think our successes are mostly in the experiments that was lost. What do I mean by that? I think a big role of experimentation in the company is obviously around risk management. 

[00:15:44] In a group like Accor, we’re pushing new code live every month and we have a pretty large development team. So whenever we make a new decision, there’s a risk for the user. There’s a risk for the experience. There’s a risk, therefore, for the business. We’re not always trying to see if what we do is actually better. We are actually sometimes just trying to see if there’s [] what we have achieved so far.

[00:16:14] And many of the experiments now that we’re launching is making sure that yes, it does solve a user pain point, but we want to make sure it doesn’t hurt our business KPIs. 

[00:16:25] And, I can give you a fun examples that, that we’ve run this year. For example, if you think probably the most important page for us is our product page.

[00:16:34] We call it hotel page. That’s where people are supposed to come and discover hotels and the different prices, etc. Obviously, we spend probably half of our experimentation efforts on this page, on those concepts. And if you think of a product page and especially hotels, we quite often make the assumption that images are really important.

[00:16:59] When you look at any kind of research, I’m sure when was the last time you went to a hotel vacation without looking at pictures. Probably never. So for us, we were always trying to push forward our hotels through pictures and we had uncovered on Agoda and on Booking.com that they’re using mini images.

[00:17:23] So when you land on the page, you’re going to see four or five mini images that are going to lead to a gallery of pictures. So we tested that variation. We thought it was a no brainer. We went from one picture to five pictures and we thought it would help users better picture the hotels straight away.

[00:17:43] And what was really funny or not so funny actually, is that this test, yes, we did increase the volume of users checking our hotel galleries. However, we drastically decreased our conversion rate. So we tried to understand why by looking at session replays and basically by encouraging people to going to our galleries. Because our picture galleries were not that good. And sometimes we had up to 50, 60 pictures. 

[00:18:13] Well, we lost people by just swiping, swiping, swiping, swiping. Basically they got tired of looking at pictures and they would leave our website. So you know, it’s one of those fun learnings where you realize you have to stress as many things as possible, especially when it impacts key user behaviors. 

[00:18:36] Because, obviously, the idea was good. Encouraging people to view images. However, when you do so, you better make sure that the experience that you’re pushing is then the correct one. So that’s one of the really powerful learning that we got. And thank God we didn’t push it live, otherwise we would have lost millions when you think of the size of our business.

[00:18:58] Those are the type of things that we’re starting to realize are super important. And also, because of using experimentation and a bit of analytics, we’re starting to realize also how certain behaviors are important for our users, such as, for example, comparing. Sometimes we too often focus on trying to show something that is pretty.

[00:19:18] But then sometimes you realize some UI that are pretty, are not efficient enough. So one of the things that we know about our users is how important it is to compare hotel rooms. Some of our hotels have over 30 types of rooms. So imagine having to scroll one by one to try to compare them.

[00:19:36] So that’s one of the things that we’ve learned with experimentation. That is a fantastic thing to have this kind of tools, as a company, and those are the biggest learning that we’ve got. 

[00:19:49] Mareen Thomas Cherian: Learning more than the fun part. I mean, images are the nucleus of this industry. Every traveler first looks at the images and this has been, I understand what a great learning experience it would have been for you as a company and then to improvise on that.

[00:20:06] So how do you measure the impact of the experimentation efforts on business performance such as revenue, guest satisfaction and employee engagement? 

[00:20:17] Alexandre Suon Perhirin: Yeah, of course. There’s plenty of things. So I think first of all, I would say at one point we tried to measure too many things and it got over complicated for the team.

[00:20:31] So we set ourselves two KPIs to measure. So we have a group OKRs. So the group works with the OKRs and we have three in the team, which is delivering a certain amount of A/B tests and also being at the initiation of a certain volume of A/B tests. So, for example, my goal is to have 140 A/B tests developed per year, and we want to propose to the product and business about 40ish A/B tests per year.

[00:21:08] I know it may sound very simple and maybe not so logic because obviously our goal is not to build as many tests as possible. But when you think about it, when you’re a team of nine people, you have to prove to your company that there is a reason why you’re nine. So that’s why I like to attach a number of A/B tests to it.

[00:21:26] It’s just to say, well one person is responsible of about 10 to 15 tests per year. That’s kind of one of the rules that we’re giving ourselves. Having said that, how do we measure the success of A/B tests? We try to keep it super simple. I had a conversation with our chief officer not so long ago about the complexity that it is to link revenue gains to A/B test. And I felt like spending too much time on it, it’s more waste of time than anything else. 

[00:21:59] Our websites are very complex. There’s plenty of, like you mentioned before, use cases where user can come from one page, from another countries to regions.

[00:22:10] There’s so many complexities to it. We prefer to use A/B test [or] experimentation as a tool for decision making. So we look at is that change on the website making the desired the change in behavior or not. 

[00:22:27] So we are not going to look at the impact on revenue. Even though we do write business case, but it’s just an indication.

[00:22:36] So we might say this A/B test has the potential of earning 6 million extra euros, but it’s really just to help the business understand the potential value behind implementing the A/B test. So that’s something that is really important. Otherwise, if we go into a bit more details, like I said, we look at, what kind of behavior we’re creating on the user?

[00:22:58] So for example, if we have a gallery of one image vs five image, we look at how many clicks there were on the gallery. If it increased where those went from? Did we lose them from maybe a click on something else? And are the people then using the gallery properly and then is it going in the funnel that we expected?

[00:23:20] So other than checking the price, other than adding a room, other than adding options so we keep things really simple. Even when we talk about the mathematics behind it, we don’t try to go too complicated by using multiple frameworks and multiple ways of calculating things. 

[00:23:41] Personally I’ve always been a fan of Frequentist because that’s what I’ve learned. So we use the Frequentist method. And based on the importance of the risk of the test, we increase or decrease our statistical significance. So I’m one of those guys who’s not afraid of going down to 80% or even 75% certainty on the test.

[00:24:06] If it’s a small thing, or if it’s something that does not have that much risk, then if we are 80% sure that we should go for it, we go for it. 

[00:24:15] Recently we started using a cupid method to have a faster results on our test. And we’ll still experimenting with that. Having talked to other managers in experimentation from Bol to global. So large organizations as well in Europe, they’ve used cupid as well. So that’s something that we’re testing at the moment. 

[00:24:40] Mareen Thomas Cherian: Accor hotel has so many different brands and you know, there are so many pages in multiple languages, and still the anthem for experimentation remains simplicity.

[00:24:55] I’m amazed by the concept itself and how you have been able to do break down such a complex thing into such a simple aspect of experimentation. So that’s a great example. 

[00:25:07] What are some of the biggest challenges while doing all of these things that you have faced in conducting effective experimentation and how do you overcome them from your experiences of breaking down complexity into simplicity?

[00:25:23] Alexandre Suon Perhirin: Yeah, of course. So something that you have to realize when you’re when you work in a group like Accor is that we’re not digital first. Yeah and that’s something that is really important. When I have some juniors sometimes arriving at Accor when I speak to other persons working in very large groups, especially in France, because you know, lots of the big French companies are not coming from a tech background, right?

[00:25:45] When you think of [], they’re coming from close. If you think of [] or all those companies, they’re not really digital. They’re coming from a product. So something you have to realize that at the top of the food chain in those groups are not digital. The top of the food chains are going to be the board and then it’s going to be many things else like distribution, etc.

[00:26:09] So you have to realize digital is not at the top of those companies and therefore we’re never going to act like a Booking.com or an Airbnb or things like this. 

[00:26:19] That’s just something that you have to realize. So from that point what’s really important is to stay focused. I’m constantly having this debate with other experimentation managers, of the difference between a strategy and planning and that’s the place that we have to discuss with products product that are going to try to plan quite in advance that’s how they supposed to perform.

[00:26:48] I’m not saying it’s the perfect thing to do but it’s something you realize in large organization is that product, they want roadmaps. They want to tell you how many deliveries they have per week, per month, per year. And that’s how they sometimes measure the success.

[00:27:02] Experimentation is quite different. Our goal is to solve problems. If you want to keep things simple and how we try to do a good job at what we do is trying to understand, well, what’s the vision behind each of those products.

[00:27:16] And that’s why I keep telling all the analysts and all the people that I can, if you want to keep things simple, focus on the product vision, focus on real user problems and that’s when you’ll uncover things and you know, sometimes i’m making that same mistake of trying to prove that we’re running lots of tests to my managers because that’s a good outcome. But unfortunately, that’s when we start doing some bad things. I think experimentation in large organizations that are not digital first the program itself has to be very, very much controlled.

[00:27:52] You have to make sure that everything that you’re pushing into, as an A/B test, has some kind of priority for the group. And I’ve learned that by mistake. Two years ago, we tried to open A/B test to everyone in the group. And so we skyrocketed our number of tests but most of them were crap, like we kept telling things like, hey we’re going to test that. We would ask someone, the stakeholder, why are you testing this?

[00:28:21] And they’re like, I don’t know, but that’s why I’m testing. Experimentation is not about throwing shit at the wall and trying to see if it sticks. It has to have a strategy. And so what we’re trying to do now is to have a really, really good strategy on what we’re supposed to experiment on.

[00:28:41] And at the end, that’s how we keep things simple. I think it’s supposed to be very clear to everyone what we’re trying to accomplish. We try to be extremely transparent. And you know, I’ve learned a lot from other people to accomplish something like that. You know, talking to, for example, the guys at Microsoft they used to tell me they have big rocks every quarter or every semester they decide on the big rocks to work on.

[00:29:07] And I was talking to somebody in Asia from Zalora and she was telling me they don’t call it big rocks. They call it moonshots. Why are the moon shots? I guess it’s a bit more cultural like that over there. I think she’s based in Singapore, Hong Kong or something like this. And that’s what I realized the best companies are always really, really focused.

[00:29:32] So in France, at least in France, we don’t have a particular name for it yet. I’m trying to find that word, but basically we sit down with the top management. So we meet the chief officers once a quarter and we decide on our big rocks and usually we have two or three projects per quarter or per semester.

[00:29:51] And so like that, it’s really clear for everyone. So once you have a strategy, then you have the core to plan how you’re going to accomplish that. 

[00:30:00] Mareen Thomas Cherian: One of your previous answers suggested that experimentation is definitely a fabric of the company now, of the group now. So could you please throw a little bit more light on the experimentation culture, and how people have adopted, how [did you get] go ahead from the management, and what does the culture overall look like?

[00:30:19] Alexandre Suon Perhirin: Yeah, sure. So I like to be very transparent on this question. Again, to give a bit of background, there was a time where I thought big organizations never had to convince anyone on experimentation one of the biggest thing that I faced when I started experimentation was actually to convince people to A/B test.

[00:30:39] And I thought it was completely logical. I’m giving you a tool to making sure that what you’re doing is correct and you’re telling me you don’t care! I was shocked by it. And to you again, just to be transparent at Accor, even though I think we’re extremely mature, not everybody wants to test.

[00:30:57] And that’s something that is important to recognize. Especially when even though you have the management that agrees with it. First of all, sometimes you cannot test things because you have limited resources. But in a larger organization, like I was saying before, there’s lots of people coming in, [going] out.

[00:31:15] There’s also sort of transformations on how teams are structured. So sometimes it does change the way you think, the way your team is placed, etc. So, I think what I can say is experimentation is definitely now sticky at Accor. So it means that even if tomorrow I disappear or my team disappears, there’ll be someone else replacing me and continuing our job.

[00:31:38] So that’s already a big win and the management knows about testing. They do roadmaps with us, etc. So that’s something that’s really impressive when you see those goals are part of the experimentation roadmap. So from a management perspective, it’s all good. 

[00:31:54] When you think of the operational, like I was saying, some teams are a lot more mature than others. 

[00:32:01] Our teams are more mature. The most mature teams is really depending on the product that they’re working on. If you think of a booking funnel where most of our users are, it makes sense that they are extremely mature in terms of experimentation, meaning that we have co-joined roadmaps, and we work together on what’s going to be tested, what’s going to go on into production, what’s not going to be tested, etc.

[00:32:29] What’s going to be wrapped in a feature flag, etc, etc. Some of the teams, we’re still trying to convince them to test, and some of those teams, we have to convince them to test, because maybe, well, the KPIs of the products are not as easy to understand as the booking funnel.

[00:32:46] You know, the booking funnel is really easy. You look at conversion rates, you look at your eCommerce revenue, etc. You look at the transition from step to step. If you think of maybe the ‘My account’. How do you optimize my account? What kind of KPIs are you looking at?

[00:33:04] Are you looking at how many times a person looked at their points? I mean, how do you A/B test that? So sometimes also A/B testing or experimentation is not the best strategy for those teams. So we haven’t worked enough with them yet. So the things that we’re doing to improve that is for reading internal audits.

[00:33:23] So I like to send a questionnaire to all the stakeholders once a quarter to understand how do they perform? How do they create hypothesis? How do they optimize the product? And if we see an opportunity to do better then we try to find solutions. 

[00:33:41] We try to grow our team as well more and more. I think having experimentation coach or experimentation ops or evangelists, I’ve heard all the names, is really, really important. Some of our tribes or squads don’t have less experience using data. So it’s important to have that people that are helping them along.

[00:34:07] To answer that question, some of our teams are extremely mature and there’s not much that we need to do except keep working really well with them and finding opportunities. Some others you know, we’ve been training for months, sometimes years, and maybe I’ll take years again to convince them. It’s not so much about convincing, but it’s about trying to understand where we fit in the ideation process and something that I said to a talk not so long ago, I think what’s really important to organize as an experimentation manager is that there’s not one way on how to do experimentation.

[00:34:43] There’s not a single solution on where you’re going to fit in an organization, especially in large organization. There’s probably 10-20 ways on where expansion is going to fit in the different teams. And your job is to crack every time for each team, how and where you’re going to fit.

[00:35:02] And it’s really changes based on the strategy that this product has, the product itself, but especially the people. Based on the people that you’re having in your team and the people in front of you, the way you guys are going to work together is really changeable, if that makes sense.

[00:35:22] So, so yeah, I think with a good culture, we can definitely do better. We’ll do better. It’s going to take lots of time. 

[00:35:29] Mareen Thomas Cherian: On a lighter note, what books are you currently reading? If you are not a book person, could you share with us some web series that you’re currently watching? 

[00:35:39] Alexandre Suon Perhirin: I have a plenty of books that I’m reading.

[00:35:41] I’ve always brought a few books with me. 

[00:35:44] Mareen Thomas Cherian: How do you distribute time to manage to read each of those? 

[00:35:51] Alexandre Suon Perhirin: I love it. I think it’s really important. I mean, not that I want to compare myself to any of those big guys, but I don’t know why I was watching a few documentaries, one about Bill Gates and he was saying how many books he was reading per year and how he was learning about it because of them.

[00:36:09] And, I just tried on an airplane when I was going to India, actually, and I was like, oh, okay, I can learn so damn much more if I start reading. Because I was used to read all the LinkedIn posts, etc. like everybody’s doing, but it’s true that books, this, it’s a different kind of knowledge.

[00:36:32] And so at the moment I’m reading all the books that are referenced by Harvard Business Reviews. Usually they’re always really good and written by authors that actually know what they’re talking about. Not so much theory, but lots of what people have experience so far. 

[00:36:50] This one is really good. It’s called ‘Innovate with Success’ and it’s about developing your creativity and giving life to your ideas. Sorry, I’m translating from French, but there’s plenty of other authors that have written on it and it gives you good ideas on what you’re supposed to do as a leader, what you’re not supposed to do. So it’s really, really good. 

[00:37:15] I always show to everyone, so it’s called Statistical Method. It’s super important. Whenever I don’t remember something, I check in it. It’s kind of like a Bible. Something that everybody’s read is the Trustworthy Experiments. And I’m reading also all those things. Quite often, I like to say experimentation is a lot about being a salesman.

[00:37:38] You have to convince people about doing something that they’re not currently doing. If they’re not currently doing it, it’s because they probably don’t want to do it yet. So like one of my team members that he offered me this book, it’s called ‘Surrounded by Idiots’ not so long ago.

[00:37:56] And I haven’t finished it yet, but it’s about the four types of human behaviors. At one point, if I give you the story, I was stuck with someone. Basically, I couldn’t convince that person to start running tests and, I was really frustrated by it because it was weird.

[00:38:14] It was the first time I was like that blocked. And every time I was convincing that person, a couple of weeks later, that person would basically stop agreeing on the process that we had agreed upon. So I was super frustrated and that really helped me understand how I can better work with other co-workers.

[00:38:31] So that was cool. I could talk about all of them ‘Working Backwards: Ways and Insights from Amazon’. That thing is incredible. Amazon is a fantastic company. The way they’re working doesn’t apply to a lot of companies but it’s good to have ideas.

[00:38:51] You know, one of the things that I’ve learned from it, if you think about experimentation, you spend lots of time in meetings, right? Explaining about how to collect a hypothesis or for example, how to share results and the thing is that you don’t have a limited amount of time. I think we’ve always had this experience where we feel like in one day, we have way too many meetings and this book really helped me build experimentation plans. 

[00:39:19] They supposed to help us reduce the time in meetings. So something that we are ‘testing’ at the moment is instead of having a meeting where we show the results with everyone, we just send the results as a test plan and we try to understand if people can just read the test plan and say, okay, I got the results. I don’t need a meeting. 

[00:39:37] So that’s something that I got from this book and I’ve got plenty of other things, but I’m like, I’m not going to spend the entire hour talking about all my books.

[00:39:50] Mareen Thomas Cherian: Awesome! Amazing! So ‘Working Backwards’ is like backward integration. If this is the result we want to achieve, then we work back towards the hypothesis. Okay. This should be the hypothesis. 

[00:39:59] Alexandre Suon Perhirin: Yeah, definitely. And there’s so many frameworks that they’re sharing. You need to help you think that way. Our chief officer, digital and business officer, she’s from Amazon. 

[00:40:13] So whenever I have a bit of time with her I’m lucky enough to hear some of their stories and she was at one point around the breakfast sharing how basically, before starting a project, they would write a note release. So a public release, a fake one, but what would it look like to launch this product?

[00:40:35] So let’s say the project is finished and you’re doing a public launch and you’re doing a public release. What would it would be? And I think more and more, I’m going to try to do that with my A/B test. How do you implement some of the feature, how do you see it happening and what do you see?

[00:40:52] How do you see, what do you see, I think, as the consequences on the users, etc. I think all those little things, I know it doesn’t sound like much, but it gives a lot more life to your experimentation program and I think it’s very important to have something that’s attractive to not only the users, but obviously the internal stakeholders, yeah.

[00:41:13] Mareen Thomas Cherian: So it’s like experimenting with experimentation. 

[00:41:16] Alexandre Suon Perhirin: Yeah. Yeah, you definitely should. I don’t understand the people that are not changing their programs all the time. Me, every two weeks, everything has changed. I want to get better at what I’m doing. 

[00:41:27] Mareen Thomas Cherian: Thank you so much, Alexandre, for sharing your insights with us.

[00:41:32] I’m sure the audience must have loved it and gathered great insights for themselves and their organisations from this session on ConvEx 2023 by VWO. Thank you so much for speaking with us, Alexandre. 

[00:41:43] Alexandre Suon Perhirin: It was my pleasure, Mareen. Thankyou so much!