Using Neuroscience to Create Engaging User Journeys

[00:00:00] Ishan Goel: Welcome to ConvEx ’23. VWO’s annual virtual summit for growth and experimentation professionals. 

[00:00:13] My name is Ishan Goel, and I’m the associate director of data science at VWO. Thousands of brands across the globe use VWO as their experimentation platform to run AB tests on their websites, apps, and products.

[00:00:27] I’m excited to have Marcello here, who is the head of experimentation at Sky.

[00:00:33] Hello, Marcello. Excited to speak with you.

[00:00:37] Marcello Pasqualucci: Hi, Ishan. Nice to speak with you as well, and looking forward to our chat today. 

[00:00:42] Ishan Goel: Great. So it will be great if you can describe to us the business of Sky, and your roles and responsibilities as the head of experimentation.

[00:00:55] Marcello Pasqualucci: Sure. If you are in the UK, Germany, Italy, and most places around the world, you probably know about Sky. It’s one of the largest tele communication companies around. And, we look after broadband, TV. They look after some of the greatest shows and, sports commentary in the world, including Premier League, Serie A, Bundesliga, and so on.

[00:01:23] My job here at Sky as head of experimentation is drive value through, running tests across the eCommerce funnels, which is anything across the website that helps generate revenue, generate values for our users, through helping them understand more about our products, our offers, and most importantly, the type of emotions the Sky offers to our users.

[00:01:49] Ishan Goel: Can you share a bit about your team and how is the experimentation effort, distributed at Sky? 

[00:01:57] Marcello Pasqualucci: Sure. My team is set up in a particular way, which is how I set up all my teams, where I have, experimentation managers that look after products, from an horizontal point of view, so across the full funnel.

[00:02:12] But then I’ve got specialists that support each manager through unique skills. One of them looks after any type of digital data and insight. Another specialist look after offline data because we’ve got loads of retail stores. We’ve got a huge customer service team. So we decided to focus on gathering insights from those teams and not just doing digital because the Sky experience spans across everything.

[00:02:43] So we can’t ignore the data that’s available from the people that come to our stores or the people that use our tools or call our call center. And then we have a conversion copywriter. As a specialist that focuses on making sure that not only there is an alignment in the way we talk to the users, but also supports the team and have a deeper understanding on the power of copywriting.

[00:03:09] That’s something that often is considered not important. Changing headline means nothing, but it actually means a lot. And if done correctly, it can really drive users to do a different set of actions as it applies anchoring, [and] applies priming at the moment users even before the users even jump into this the funnel.

[00:03:31] Ishan Goel: So, as I understand you have many experimentation managers and then there are skill experts. So one skill expert is helping out multiple managers? 

[00:03:43] Marcello Pasqualucci: Exactly. The support experimentation specialists, as we call them, they can then move on different products as a bigger focus, depending on business needs, while the experimentation manager makes sure that there’s an ongoing flow of experiments across all products at all times.

[00:04:06] So making sure that, we don’t move all our chips on TV and we forget about broadband, for example. It’s very impactful. 

[00:04:17] Ishan Goel: Really fascinating. So it seems like you have a high velocity of experimentation across these multiple experimentation managers. Is velocity one of your focuses in experimentation?

[00:04:30] Marcello Pasqualucci: At the moment, yes. You need to consider that even though a part of Sky had a great history with experimentation, the eCommerce platform was a bit behind. So we pretty much build the team from scratch. And so focus now is running as many experiments as we can, because it allows us to better understand and improve our internal processes, which is something that we cannot forget.

[00:05:00] And, they help us also get from a point where they were often tests breaking each other, to having no test breaking at all in the past six months. And that’s just down to the fact that we started testing more, we learn what teams were involved in releases across the journey, what each how each component across the journey performed, and work with each other.

[00:05:26] And that helped us get a better understanding of the tool and that’s an untold power of velocity, it’s about getting a better understanding of your business and of your user base while of course you start working on improving your prioritization and your targeting so that you can drive more and more value over time. 

[00:05:49] Ishan Goel: That’s interesting. So from what I can understand is that you have a special focus towards interpretability of tests and what you’re learning from the tests, so that you can improve and ramp up the experimentation effort.

[00:06:02] Marcello Pasqualucci: Exactly. And also we started adding up psychological profiling on each test to understand, you’re on an experiment, what does it mean for the user? What does it mean from their, brain? You know, there’s a [quote], I can’t remember who said that, but there was a quote that I really find important, which is ‘you don’t optimize products, you optimize brains’. And, that’s very, very true. So the better you understand how those brains work. And my background is in neuroscience, so that’s my big focus. So the more you understand how people’s brains work and how your products make their brains work, the more you can improve on your targeting and you can improve on your efforts, it changes the way you write copy, changes you think about design, changes the way you decide to push for sale or push for getting more information.

[00:06:57] So the journey itself changes alongside the user. We don’t try to force the user to just adopt what we have because that’s pretty much defeats the point. 

[00:07:11] Ishan Goel: Quite a unique perspective there, Marcello. That’s very unique at least [among the] experimenters that I’ve talked to. 

[00:07:19] I would ask you a question – velocity or the quality of experiments, which one of the trade off would you choose and why and how do you go about it?

[00:07:30] Marcello Pasqualucci: Well, to be honest, when I think about it, I try to ignore the concept of Velocity. The amount of tests you can effectively run, it’s also based on how many people can work on the test. Because anybody can sit there and run a thousand tests, but it could be absolutely meaningless. Or you wouldn’t have time to do the pre test analysis and post test analysis.

[00:07:53] So you miss out on insights and knowledge that could otherwise be shared across the business to help everyone grow. Because the job of the experimentation team is not just about testing or to improve conversions. It’s also around, create a legacy of know how that’s passed across multiple teams.

[00:08:13] And, especially because working in a corporation, you get dozens and dozens of people working on this one small journey. And if that’s not, that knowledge sharing doesn’t happen, whatever you do then becomes pointless, because then you just need a couple of people to change some of the other things that you need to start again from scratch.

[00:08:34] So, more than thinking about velocity and quality, first of all, what do we mean by quality? For me, my view is that you should prioritize the quality of the insights that you use to come up with your experiments instead of just thinking, but this experiment, if it wins, it’ll make me an extra million pounds.

[00:08:58] Well, before you run something, you have no clue if it’s gonna win or not. Even if you have loads of data, you still test because you want to make sure it’s the right decision. So, that’s why, if your experiment is based on data from research, analytical tools, heat mapping, session replays, call center calls, interviews, surveys, they’ll be more valuable in the long run.

[00:09:30] I’m not talking about one or two tests. I’m talking about 200, 300 tests. The test that you prioritize that way will be more valuable than the ones of, okay, I woke up this morning and I thought that that would be a very cool thing to test. Anybody can sit around the website and come up with that type of test. But you need to look at the data to be sure it’s worth it.

[00:09:50] And, one of the things that we look at is, win rate. And we over the years, I found that there’s a bit of a Goldilocks zone between, 25 and 45 percent which is usually where we aim. I always try to aim to be because it tells me two things. First of all, that I’m going in the wrong direction.

[00:10:13] So at least a third of my experiments are bringing value to the business. And also they are not playing it too safe, also trying to innovate and 10 different things. More importantly, one thing that we do a lot is as a requirement for each experiment, we request two or three variations. We don’t run A versus B.

[00:10:35] The idea behind it is that the more variation you test, the more ideas effectively you’re testing. And you’re more likely to find the winning solution. By default, the statistic, the more things you test, the more likely you find the right one, as long as the data behind it is good.

[00:10:57] Ishan Goel: You mentioned that you fuse your passion for neuroscience with your experimentation effort. I would want to know more about that on how you use neuroscience principles in the customer and user journeys that you designed for Sky. 

[00:11:12] Marcello Pasqualucci: One thing that I’ve started doing the moment I joined Sky was actually running lessons for all the teams.

[00:11:19] So the teams at Sky have access to my knowledge and background. Every two weeks, we do a one hour lesson. Anybody can attend. And we’ve got people from interns to directors coming to the sessions to learn a bit more about neuroscience and behavioral science. And then that also help us understand more about the user behavior, understand why people are doing something.

[00:11:48] And that’s very important because the job of experimentation is not about finding answers. We are not the ones with answers. The users have the answers. Our job is about finding the right questions. And you can only find the right question in two ways. Either you ask loads of questions, and that goes back to the loads of random tests that sooner or later will find the winner.

[00:12:09] Or you understand what the problem is for the user and then start creating formulating the right hypothesis and the right question for users to answer for the experiment. And, so the use of behavioral science in what we do is, something that the whole team embraced. First of all, they find it very fascinating, very interesting and stimulating, even.

[00:12:33] And, we are trying to also track which principles are being applied across each experiment. And then try to map out, almost like a brain map. If we do this experiment, hypothetically, we are figuring this zone of the brain, which means that users will potentially feel these emotions, which can then be associated with this product.

[00:12:58] Is this the right emotion for this product at this stage of the journey or not? And we actually build emotional maps. For our journey to understand what people feel step by step and then decide, is this based on surveys, we know users are feeling confusion on this step while effectively want to make them feel.

[00:13:17] Say if you want to make the feel content. And then we run experiments in changing that emotion. You can imagine what changing one emotion might take, one experiment might take ten. All different ones because they impact different components across the step of the journey. And so on. And that’s actually quite interesting when you look back into things and you realize actually we change his emotion that impact the revenue this way. So then we can go to the business, Sky, where our business is not about selling products. Our business is really about selling emotions. 

[00:13:55] Ishan Goel: Such a unique perspective. That’s very, very interesting.

[00:14:00] This concept of emotional maps, it’s really intriguing and very rare and unique to the industry. I want to understand how do you optimize and personalize the digital experiences for different segments of your audience?

[00:14:16] Marcello Pasqualucci: We try to look at what’s the best default.

[00:14:22] You always need to have a strong default journey, a strong default pages or experience because that’s what runs in the background. That’s what tells you at least we’ll make this much money. Because that’s what companies want, money. When you start looking at personalization, then you start to consider also the other tools involved.

[00:14:46] So, what’s the quality of the data we collect? What’s the type of data we collect? What can this data tell us about the user? A while back I wrote this article around tracking weather patterns. For eCommerce optimization and personalization because it has a huge impact. Just think about it this way.

[00:15:08] Yesterday, my wife went out. Yesterday was raining in London. It wasn’t sunny today, but it was raining. She ended up going to a store with her sister and she decided to buy fondue set. Now, if it was 30 degrees outside, she’d never buy a fondue set. But, if you know that’s the pattern. If you know that the weather is actually influencing what people feel makes sense for them to buy, then you need to make sure that you prioritize the right product based on these type of factors.

[00:15:50] And, weather has a huge impact. Even the day of the week. I think when you talk to people, especially if you’re trying to get their, support, we try not to have those meetings on a Monday. You want to do it on Tuesdays, Thursdays. But also you don’t want to do it on a Friday, because Friday people are trying to shut off.

[00:16:12] They don’t want to listen to any problem or any issues. They just want to get on with their day and then have a nice weekend. So you need to take into consideration things that are not exclusively digital. They have a big impact on the way our brain works. If you look at also things like the way we use our devices.

[00:16:33] I remember doing this, looking at this project when I was in banking, where there was this provider that could tell us, in half a measure the angle at which users were engaging on our banking app. Based on that angle, we’ll be able to then understand if users were more open to suggestions about new products, or they felt concerned about something. So the phone angle was different than their average [angle] closer to them, and that would tell us if they actually needed more support and more safer environment to do their operations.

[00:17:12] You can imagine that changes your whole strategy. But those are things that often are either misthought or misunderstood. Because we’re so focused on the little bit of data that we measure that we don’t realize that it’s just the tip of the iceberg. What’s available in the online is not what’s available in the rest of the world. 

[00:17:31] Ishan Goel: Like the kind of factors that you’re talking about to personalize, they for an online visitor, they must be very hard to track, so it’ll be great if you can tell me how do you track those variables and how do you know that a visitor, belongs to a particular segment, or this is the personalization we want to do today. How does that work out? 

[00:17:51] Marcello Pasqualucci: Well, weather is not hard to track. It is a much easier thing to track.

[00:17:55] And if anybody interested, you can just go on my profile on LinkedIn, and there’s an article that explains how to track it on Adobe Analytics and a link to another article that explains how to track in Google Analytics. I don’t know if it’s still okay for the new Google, but at least it gives you an understanding of how that’s done.

[00:18:14] And that’s already one of the options.

[00:18:16] The other thing that you should keep in mind is that there are almost preset patterns. So we need to also start looking at understanding what behavior on the page matches this specific behavioral pattern. And it takes time.

[00:18:38] That’s why, it’s not something you can do on the fly. It takes a few months, if not years of work on your data and your profiling to really understand what’s going on. Sometimes you’d like to run testing to really be able to understand that that’s a specific issue or that’s a specific user profile.

[00:18:58] And it’s really worth your time because besides the fact that it’s fun and interesting. But building a behavioral map on your website is now, you almost have to. Because we’re going towards a copious world. So you won’t be able to track a lot of stuff. You better start thinking about the future now, because you don’t want to risk it.

[00:19:24] You want to make sure that to be able to understand who those users are just based on the way they are working on the website. Think about the way we use the mouse, the way we type. It’s a unique trait the way you type is different than the way I type. So it could actually be used to recognize you as a customer, and automatically log you into a profile. There’s lots of things that are out there. Do the research on this, that the people are not really going for it because it takes a lot of work. But we don’t do this job because it’s easy. We do it because it’s hard. And, we do it because we want to know. And the more we know, the happier we are. 

[00:20:05] Ishan Goel: So seems like you are deploying some learning models on user behaviors. Are there some learning models or you create some if-then-else trees? Or how does that happen? How do you relate someone to a brain pattern? Do you deploy some data science or AI there? 

[00:20:23] Marcello Pasqualucci: We are currently looking into creating very basic patterns to understand. First of all, propensity to buy. Which is nothing complex. But at least it’s a starting point. And then we are also still implementing some other tools that will help us collate user behavior from the way they use our tools, so that we can get more and more understanding. We are still in the infancy of it. As I said, there’s a new team. Only been in Sky about a year and a half. The team has been together only since January, [or] February.

[00:21:05] So we are still on that learning path and curve to really get the best out of the data and the information we have available. But yeah, the idea is to get to the point where we have, some modeling the feeds into automated audiences, which then will trigger and send users to diverse experiences live on the website. 

[00:21:33] Ishan Goel: I’m just imagining neuroscience and experimentation has such a great depth into it. That’s a great perspective, Marcello. Can you discuss what innovative testing methodologies do you use in your experimentation efforts that has a short term or a long term impact, apart from what you just shared?

[00:21:54] What are the other innovative methods that you use? 

[00:21:57] Marcello Pasqualucci: Basic methodology is always the same as a scientific method. You measure, hypothesize, size, test, and measure again. And that’s all you do. Day in, day out. One thing that we’ve been introducing, though, is a different way to come up with innovation. Because testing, if you look at experimentation, it’s effectively like a highway.

[00:22:22] You’ve got your basic front end and then testing. You do loads of those. And then knowledge from those tests, feed into the mid lane which is your bigger pieces of work, where you start looking at, new functionalities and things like that. Now, all that knowledge feeds into the good fast lane, but effectively is that the lane where you do probably a lot less experiments and, they take a lot more work and a lot more insights.

[00:22:51] But effectively, those are the ones where you actually change the whole journey, because sometimes there’s just so much you can test before you realize, actually, we really need to make a big step change here because the journey is old. The journey maybe was set up incorrectly. Users have changed. 

[00:23:09] If you think about certain aspects of our eCommerce and commerce world, there are sometimes connections you don’t expect. For example, what do you think was the industry that was more impacted by the release of the iPhone? They had to completely change the way they work.

[00:23:30] Which one would you think it is?

[00:23:32] Ishan Goel: Is it the Android industry? No. 

[00:23:34] Marcello Pasqualucci: Handcreams used to be all based. On a smartphone, all based creams will leave a trace. First of all, they will make it dirty very quickly. In some cases, even damage be the screen, especially the first type of smartphones. And they look terrible in selfies.

[00:23:53] They have to change now, like you look at the hand creams now, the vast majority are water based, which don’t have the same issues. Even though they’re not as effective as oil based creams. And that’s [something] that has nothing to do with smartphones or tech. They have to change to support the needs of their users.

[00:24:09] So you need to keep that in mind. Because sometimes you need to make a big decision and big changes to support a different industry that’s decided to completely revolutionize itself. And when we look at that, we go back to the basic elemental thing that Sky is, which is emotions. So, as I say, for me our emotions are very important.

[00:24:35] So, I’ve introduced this process from filmmaking. In filmmaking, Alfred Hitchcock used this Green Book, Blue Book process. The Green Book simply listed how the cameraman had to move, what the timing was for each scene, what actors had to say, and then the Blue Book would describe each scene by scene, what emotions did he want the users, the viewers to feel.

[00:25:06] If you use that in digital, and you flip it around a bit, you can actually say, Okay, this is my eCommerce journey. These are the emotions I want to use this field step by step. Now, what are my components? What are my, features and tools that I have available that help drive those emotions? How will they look together?

[00:25:25] What other information can we use to make those emotions even stronger? And that’s how we then work with the design team to come up with brand new challenger journeys. And it’s quite important because it really helps them focus on not based on what the user wants to buy, but based on what the users need to feel.

[00:25:50] Ishan Goel: When you are designing these emotional maps, what emotion should come to the user in the part of the journey, do you hypothesize there or do you actually go on customer calls, maybe have discussions with them and then track their emotions? I’m intrigued. How do we even create this emotional map?

[00:26:11] Marcello Pasqualucci: I think the name is Irrational Lab. They did amazing research on this. If I remember correctly. They can explain how those works even better than everyone. But the way we do it is we spend time, really understand the users to research, listen to call centers.

[00:26:32] Even looking at mapping from the ContentSquare or analytics to help us really get the sense of where people are feeling frustration, which is a very bad emotion, or where people are feeling uncertainty, or when the people are feeling lack of trust because we have tools that help us connect digital experience with calls to the call center.

[00:26:52] Then we can say, okay, people were looking at the deals page, they didn’t trust that they called the call center. Why are they feeling that emotion? How do we change them? That’s how that works and you can either do it for the full journey, you can do it for a single page, you can do it for a specific component, what type of emotional change in this component will potentially trigger. Based of course on our background and knowledge of behavioral science and our users behavior, and our users, because what works for us might not work the same for Amazon or for Curries, but it will work for Sky.

[00:27:33] So always important to do the research on your product and just look at what other people are doing.

[00:27:40] Ishan Goel: That’s amazing. From what I’m able to understand, you have defined heuristics for different actions, maybe like rage clicks or dead clicks, and then you have associated them to different emotions. That’s interesting. How many emotions you usually go around with? Is there a rough number?

[00:28:03] Marcello Pasqualucci: What we look at is around 12 different emotions from very positive to very negative. Very positive emotions are just as dangerous as very negative emotions. But you need to understand what they mean for you. So what I see is 12 other companies might be fine with five, depends on how detailed you want to be.

[00:28:25] And, in some cases we will look at very basic three emotions – happy, content, unhappiness. That’s it. Or you can go into more details like, terrified, unclear, doubtful, trusting, it really depends on how you want what your website is, what you’re trying to sell, what you’re trying to do for your users.

[00:28:52] But the visualization of it is what I always find interesting because it effectively it is like a line report that shows the emotions as they change across the journey. And then you can say, how do we change these emotions to the one right above it, which is slightly better. And then for each journey what we’re planning to do next is to understand how this changes emotion, then impact, and so on.

[00:29:20] But it’s gonna take a year or two before we get it right. At least to the standards that I’d like to get to. 

[00:29:28] Ishan Goel: That’s very interesting and very interesting discussion. I think our listeners will be getting the feel that they’re in the matrix. But I’ll move further. I see that you also do coaching sessions and dedicated one-on-one sessions.

[00:29:45] How do you ensure effective team leadership and knowledge sharing across the teams with those sessions? I would like you to throw some light on that. 

[00:29:54] Marcello Pasqualucci: Within my team there’s two team sessions per week. One of them is just about testing and the other just, what are we looking at, what we want to do, how do we support each other, which is very important.

[00:30:07] Again, we have a bit of a special matrix here within the team members. So people never work effectively in isolation. Then we have a team meeting where we simply talk about team life, company life, any updates from the directors and so on. And we really look at how happy are we. Because happiness is very important.

[00:30:33] A team that is happy and sticks together will always outperform a team of geniuses that can’t talk to each other. And, then we run around weekly one-on-ones with every single member of my team. I always try to focus on how they’re feeling at Sky but also any corporation.

[00:30:55] There are loads of training on this, so there’s loads of tools available, but we really try to understand where they are now, where they want to be in future, how do we get there, what type of knowledge that they need to see, what type of knowledge we can transmit to them, and how can we help them get to it.

[00:31:14] And then we do two “classes”. Three classes every two weeks. One of them is that we call it the state of play, which is everyone in the company can come and share every test that has been launched, every test that’s live, and every test that is completed and what it means for the business.

[00:31:39] There’s another session which is my behavioral science classes which are on a Friday morning. Very relaxed. When they started, there used to be a lot of people from digital. Now its a lot of people from the research teams are coming in to better understand, behavioral science and how they can use them for their business.

[00:32:03] And, then another one which we launched recently is Lunch and Learn. So lunchtime on Monday, we have a room booked, it’s always the same room, where we sit there, as we have lunch, we actually do a data review. So we look at funnels, we look at ContentSquare data, we discuss calls, you can listen to live calls from the call center, and those are open to anyone who wants to come and join, because the idea is that… a great idea can come from any mind, as long as they have the right information. So we try to really open it up, but also help us teach other teams how to use some of our tools that we use in a way that is more effective and more targeted. Because it’s easy to gather data. It’s very hard to turn that into insights if you don’t know how to do it.

[00:33:01] Ishan Goel: What I’m able to understand that this will be helping you to build the experimentation culture in the whole organization also to quite an extent.

[00:33:10] Marcello Pasqualucci: Exactly. You need everyone’s buy-in, everybody’s involvement, because there’s so many teams involved. Our Sky campus in London has our own bus routes, and we have six buildings with thousands of people in each building.

[00:33:29] In the building I’m working in campus, we got our own restaurants, four coffee shops, our own supermarket, and post office. Within an office. And that’s not something you get. We have a cinema, interestingly enough. So it’s a big company. And, you cannot sort that out by sharing a report, sending an email.

[00:33:56] It doesn’t work. You really need to get people to sit there with you and get excited about what you’re doing. Because they have to understand how what you’re doing will make their job easier and make their job better and more exciting. If I think about my time alone, it’s about probably at least a day a week, just about looking after the team and looking after the business.

[00:34:20] And, you can imagine that it takes a lot of effort. But in the long run, it’ll be worth it because the more people understand, the more people learn, the higher the quality of inputs and support we’ll get from the business over the year. 

[00:34:38] Ishan Goel: Yeah, and the whole effort compounds in the process over time.

[00:34:45] So I want to ask you what are the common challenges and obstacles that you’ve faced in your experimentation and personalization journey at Sky? And how do we overcome these common challenges and obstacles? 

[00:34:59] Marcello Pasqualucci: Well, the main challenge, for any company or corporation is really about, we’re going to experiment, what does it mean in sales or why can’t I see it in our reports?

[00:35:10] So it’s really about, understand from one side, we experiment because we want to know. It’s a knowledge work. We are scholars. We are not marketers. At least that’s my view. While if you look at the business, the business of course wants to know, I’m spending X amount of money on you and your team and your tools. What’s my ROI?

[00:35:34] And you can’t ignore that. You need to make sure that they get something back. Something is visible. So that it’s important to understand what are the numbers you’re going after without obsessing over it? What’s your focus in business? As I say, the way we work allows us to not have a specific focus for the experimentation managers, but then we’ve got the specialists moving around, increasing the quality of the work done on specific products, over time, so that helps us hit some of the targets we have. Then spend time with your financial teams and your trading teams, and your friends, and not your enemy, but in most cases do not understand how your work, how you work. For example, one of the questions you get sometimes is, which sparked a few discussions in the past was, if you’re running an experiment, we need to know because it might impact our trading reports.

[00:36:37] If my test, which is on a portion of the users coming into the journey impact your trading reports, let me know my uplift. Based on the calculations we’ve done, we’re looking at 50, 60, 70 percent uplift. For this number, this test have such an impact that would change your general trading reports, which includes stuff that we’ve got nothing to do with. Which is this offline data as well.

[00:37:06] But that takes time for them to understand because, of course, they want to make sure that any changes map to specific activities. But you cannot make the job hard, so you need to find the common ground there. Also there are financial teams, it’s often you want to report how many sales your experiment has generated, but if you have a big funnel, you have a test up here, even if there’s a significant impact on sales at the bottom, you need to be sure that they understand how that’s calculated and what it means.

[00:37:40] So, we work with them to build a bit of a calculator to have an estimate of the financial impact of each experiment. But one thing that we’ve done, though, is we had the deprecation, monthly deprecation impact on our winners, which from what I’ve seen online should be around 2.5 percent average in eCommerce.

[00:38:00] I would love to spend more time to calculate ours, but then we’ll take it forever. So we just stick with an average. So when we calculate the impact of our experiments, we look at, month one it’s an extra a hundred sales. By month twelve, we’re looking at 78. Because that’s the impact over time and how that deprecates and changes.

[00:38:23] That help us also get some more confidence from the business on what we’re reporting makes sense or not. And then, of course, you have to understand the mean impact of your experimentation. Because if your company does one million orders or sales per month and your [] uplift does fifty thousand extra sales, chances are you’re not going to see it.

[00:38:45] You’re not going to see the jump in the data. Now, if you run 10 experiments, and 10 experiments get an extra 50,000, and you put them together one after the other, you put them live, and you start seeing the impact because hypothetically, again, it doesn’t happen, but hypothetically, you should see about a bigger impact than one single experiment.

[00:39:07] So then you start seeing a three, four, five percent of increase in orders, I can visualize it. That’s also where then velocity becomes important because, either your velocity or your impact need to be great enough to impact those trading reports. Otherwise, nobody’s going to see what you’re doing.

[00:39:27] Remember, especially if you work in a corporation, there’s a lot of noise, a lot of teams doing stuff. You might have TV ads going out here in that corner. Then there’s the digital ads going out there. And, they’re all doing their own thing. And it’s very hard to connect all the dots.

[00:39:45] That’s why sometimes you need to ensure that you’re really focusing on the important things, but also you are running the right velocity to show the impact of what you’re doing. One test a month, unless you’re doing a full redesign that has a huge impact, chances are nobody’s gonna even know you’re there.

[00:40:04] Ishan Goel: Yeah, learned a lot from this session. I have one last question to ask you. What books are you currently reading? And if you have some recommendation for our listeners. 

[00:40:17] Marcello Pasqualucci: Books, well, there’s a loads of them to be honest, all such and not to focus only on business books. For example, one of the books I enjoyed the most which I used to set up my team is called, How to Run a Country, from [Marcus Tullius Cicero].

[00:40:36] And one of the thing that it explains, it’s also connected to some of the things we do in my team, my team still doesn’t realize it, or probably I have to explain to them one day, but the way I set up the team is, we set up some of our tasks, which are not about experimentation, but around our team life, our communication are based on the way that all the major religions are created. Which is to create a successful country, successful religion, you need to create a sense of belonging to create the right storytelling, the right rituals, symbols. We actually have a logo for my team.

[00:41:17] You need to create a vision, a sensory appeal of what you do. That’s why they getting together is very important for my team. We are one day a week where we are all together in the office. You need to create power of the enemies, it sounds dodgy, but you always need to create a tribe that is different than another tribe.

[00:41:38] We work with everybody, but there have always been somebody you see as a bit of an issue. You create an evangelism, so you go out through the business and tell what you’re doing and how you’re doing it, why you’re so great at it. You also create a sense of mystery. If you think about how some of the things that we do, some of the ways we work and around the psychology of the users and so on, it’s very mystic and mysterious for the rest of the business, which creates a sense of holiness around the team and also a sense of grandeur.

[00:42:12] I always tell my team, when you work into a meeting room, people need to know you’re there. You need to be seen, you need to be heard. You need to be available. You need to be somebody that people look up to. For example, by chance I heard, which is silly for me to be proud of it, two people from a completely different team talking about something, and one of them telling them.

[00:42:44] No, don’t do it that way, Marcello will tell you off. Sounds silly, but that creates the sense, that guy can come, he’s not my manager, he can come and tell me off, I need to be careful here. Which, I don’t really tell people off, but if you do something in a way that damages our ways of working, then I’m not going to just be quiet and say, ah that’s okay, we’ll survive.

[00:43:06] No, I’m going to sit down and make sure that doesn’t happen again. Because it’s the future and the legacy we’re trying to build is too important. You just let it go. So that’s the type of book I’m reading. Also, one of the things I used a lot, I often use references, for example, if you ever read the book Jurassic Park, on how they used statistics to realize that there were more dinosaurs than expected.

[00:43:35] That book really has a really good example to explain people how statistics works, in a very simple way. But business books, I pretty much read anything from, Roger Dooley. He’s great. One of my favorite people and also favorite writers. Still trying to organize a pint together.

[00:43:58] And, currently in my commute, I’ve been having fun reading, Small Data. Because, as I say, big changes, big evolutionary changes do not happen at large scale. They start very small. And learning how to recognize them before they happen. That’s what makes a difference between surviving and conquering the future.

[00:44:24] Ishan Goel: Very, very interesting. So many interesting book recommendations. And from what I understand, the first book that you were talking about, it’s by [Marcus Tullius Cicero], so it must be a 2,000 year old text that you’re applying at your work. 

[00:44:37] Marcello Pasqualucci: Actually, it’s not Marcus Aurelius. My mistake. It was from Cicero, which was also another emperor.

[00:44:44] It’s a few thousand, 1,800 or 1,900 years old. And, Cicero was an amazing statesman and was a very clever but also very understanding of his people. And, he actually built this guide. It’s a small book, but it’s a very important guide on how to govern and create a strong country.

[00:45:10] And your team is a country. It’s your country. And you cannot mistreat them or they rebel. You cannot be too light on them or they just don’t do anything. So you need to find the right balance, but also you need to create a sense of pride of being part of the team. So that you don’t have to micromanage them or look after what they’re doing, deliver a high standard quality of work without you having to ask them to do it. Which is very important.

[00:45:41] Ishan Goel: All the ideas that you shared. That’s amazing. That gives me a lot to actually delve into and find out. I’m definitely going to read this book by Cicero. I think our listeners would be enlightened very much with the unique perspective you are bringing into experimentation. We thank you, sincerely for your time.

[00:46:00] And this was an amazing conversation. I would like to wrap this up now. Thank you. 

[00:46:07] Marcello Pasqualucci: You’re welcome. Thank you.