How Cross-Collaboration Has Increased Velocity of Experimentation at Tele2

[00:00:00] Vipul Bansal: Hello everyone. Welcome to ConvEx ’23. VWO’s Annual Virtual Summit for Growth and Experimentation Professionals. My name is Vipul and I’m a Senior Marketing Manager at VWO. 

[00:00:19] Thousands of brands across the globe use VWO as an experimentation platform to run A/B tests on their websites, apps, and products. 

[00:00:27] I’m excited to have Maria Luiza with me here who is the CRO Lead at Tele2.

[00:00:34] Hi, Maria. Excited to speak with you. 

[00:00:37] Maria Luiza de Lange: Likewise, looking forward to the session. 

[00:00:41] Vipul Bansal: Awesome. Awesome. So, Maria, as I’ve been doing with other guests on the panel as well, it would be great to have a bit of a background about you as a business professional, as an experimentation professional. And also if you could just in a nutshell explain about what Tele2 is.

[00:01:01] Maria Luiza de Lange: Sure, of course. So I started my career in 2012 at Electrolux Europe, where I spent nine years. I come from the business side. Originally, I was a part of what was called the in-store team, where we were doing catalogs, leaflets. So very, let’s say, offline touch point. But as digital became more and more relevant, I started working on these omnichannel projects, that I’m sure a lot of marketing professionals are working with. 

[00:01:32] One example was we were based on our product information management system solution. We were trying to automatically generate catalogs. So that was one bridge between the in-store and the digital team, where we basically shared the database and I started working more or less as a product owner for that particular project. 

[00:01:53] Then after spending half a year in Poland, in the local market of Electrolux, where I was responsible for 360 launch of the new laundry range of appliances. I moved on then to the digital team. So that was roughly around 5-6 years ago.

[00:02:14] I started as a content manager and at the same time, I got this side responsibility which was called the analytics. So kind of here you go, there’s this baby called analytics and kind of do with it, what you please and as you can imagine obviously that very quickly became the role itself.

[00:02:37] I gradually built the analytics capabilities at Electrolux. At the time when I started, most of the resources were consultants and then gradually those roles were moved in house. Where the team that I was managing was responsible for 90 domains. We were responsible for both implementation of tracking.

[00:03:03] So working with Google Analytics and GTM, but then also experimentation and personalization. So all these areas the team was responsible for. So that’s kind of in a shortcut, the experience at Electrolux. Then I moved to the agency side. I was working for Precise Digital. 

[00:03:23] I learned a lot there about the different business models and I was working with different clients that had different setups in terms of their MarTech and, I was running maturity audits in terms of their data maturity and suggesting different data projects to run for example, consolidating all the marketing data sources in BigQuery and so on. 

[00:03:45] So very much on the analytics side and a year ago, in August, I joined the great team at Tele2, where I have been building the CRO capabilities, since.

[00:03:58] We also just got awarded as the winner in team category of Experimentation Culture Awards. We’re very, very proud of that at Tele2. So that’s just a little bit of my background, where I come from, so very much from the business and yeah, and these three experiences. So Electrolux, Precise Digital, and Tele2.

[00:04:21] Vipul Bansal: Awesome. That’s a really interesting journey. I’m really, really curious to know what was your CRO moment, the experimentation moment of sorts, the time where you realized that, hey, this is the right direction that I moved in when it comes to my career.

[00:04:37] Maria Luiza de Lange: Yeah, that’s a good question because actually when I was working at Precise, I wasn’t directly connected with CRO. Obviously when you do analysis, you always come with hypothesis and suggestions for tests to run for your clients and that’s what kind of indirectly I have been doing. 

[00:04:55] But I really noticed that I was missing, CRO and that is also why I came back to this area and really what gives me satisfaction is having this ability to directly impact positively, because we have the data, the consumer experience, and I think that’s the biggest power of experimentation. 

[00:05:16] Vipul Bansal: Could you briefly share about the history of experimentation at Tele2?

[00:05:20] And what has the approach been for the company historically? And the benefits, of course you’ve been with the company for over a year now, less than two years but what has the benefit? What are those benefits that the organization has seen from experimentation?

[00:05:39] Maria Luiza de Lange: Yeah, that’s a good question and just to mention so Tele2 as a telecom provider, so we sell mobile subscriptions, broadband subscriptions as well as entertainment, so TV subscriptions within our portfolio of products that we offer on the Tele2 and convict brands.

[00:05:55] When it comes to the experimentation, history and culture at Tele2, I’m happy to say that actually there’s quite a strong backbone of experimentation.

[00:06:05] So I didn’t come to a totally new, green so to say organization. There were already a lot of foundation and pillars were there. Now, of course as a telecom industry, there’s also a lot of kind of technical skill set if you like. So, there’s a lot of competence in the company. Also, the developers are great.

[00:06:27] Fantastic, really competent, as well as all of the main stakeholders. So, in terms of experimentation, they actually have been running before I joined tests on both, Convict, Tele2 within apps. So there has been a history of running tests. 

[00:06:45] What happened was there was a reorganization at the beginning of 2021 and as a result of that, the growth teams, as they were called at the time, they lost the development capabilities. So basically the tests were not being prioritized and that was the challenge at the time. 

[00:07:01] So when I joined, we didn’t have a developer resource and that’s what I spent the first two months together with management to ensure that we had these development resources so we could restart this entire engine again.

[00:07:14] So yeah, happy to say that, we’re now up and running experiments again. We have done a lot of work, which I kind of summarized in our application award of what we’ve done at Tele2 to get where we are and we also have fantastic people to learn from right now. Marianne is working at Tele2 currently covering for maternity leave.

[00:07:36] So we also have one of the best CRO protagonists within our organization to learn from. 

[00:07:44] Vipul Bansal: She was our guest for Convex 2022 and some really interesting insights that she shared in her presentation. 

[00:07:52] That’s great to hear. So yeah. One insight that I found in your answer is the lack of resources and how it led to just stopping, putting brakes on the experimentation program altogether.

[00:08:06] This is very consistent but what we also observe from all the feedback that we get from our customer success teams from the respective customers. Lack of resources hits heavily on their experimentation programs. So I’m curious to know experimentation sits at various stages of the entire business growth journey. 

[00:08:28] First of all, of course at Tele2, what area of this entire funnel, what stage of this entire funnel is experimentation focused on? 

[00:08:36] And also if you could talk about if experimentation is part of the product development a process as well?

[00:08:45] Maria Luiza de Lange: So, at Tele2, we have quite a unique setup in the way that we run experiments because we actually don’t have the challenge that most of the organizations have, in terms of shipping things or delivering code to production. 

[00:09:01] So the way we have set things up is that our variants are directly deployed into code base and then we just pull the variation with the experiment ID. By doing that and having this dedicated developer resource, releasing things after announcing the variant a winner is very fast for us because we have the code ready. 

[00:09:22] So, yeah, from that point of view, as soon as we have test results, we basically just inform our developer that this has been a winner and they can release it up to 15 minutes to push that into production.

[00:09:38] So, yeah, from that point of view of course, we have very quick gains so there is no this challenge as a lot of organizations talk about of leaving money on the table where they know this is a winner but they have to wait for the kind of standard the dev team to prioritize their tickets to release to production.

[00:09:55] We avoided that issue overall. So yeah, that’s definitely a privileged position to be in as a part of Tele2 group. And what I noticed is, having this perspective of two companies having worked at Electrolux and Tele2, different companies and different organization setups have different strengths.

[00:10:19] So what is a strength at Tele2 aside from this unique setup as I mentioned, is also that our UX researchers are doing a fantastic job and they are doing their usability tests themselves. So that is great! 

[00:10:33] On the other hand, what Electrolux has been really good at is whenever they had their development roadmap, planned for the year, they would ensure that all the tests they are running are connected to the features that we’re releasing.

[00:10:47] So, we’re not there at Tele2 yet. To us, kind of the excellence that Electrolux has worked out and I hope they continue. But, yeah, that’s what I’m just trying to say. 

[00:10:59] Different organizations have different strengths. They might not realize what their strengths and weaknesses are but that’s the great thing for me that I’m able to, based on the experiences across different businesses to compare and see, okay, this has worked right here. I know we can do it this way and then bring that on board and of course, everything that I learned now at Tele2, maybe in the future, I will bring somewhere else on board.

[00:11:24] Vipul Bansal: I get a sense that you have built the A/B testing platform at Tele2 right? Is that the case? 

[00:11:37] Maria Luiza de Lange: We’re using vendor platform. 

[00:11:40] Vipul Bansal: Oh, okay. But what are your views on build vs buy debate? 

[00:11:45] Maria Luiza de Lange: For us, it’s just not worth it financially. It’s as simple as that.

[00:11:50] We consulted. So obviously now, with the whole Google optimized sunset, we’ve been on that bandwagon and we’ve run an extensive tender where we looked at the offering of multiple vendors and we’ve consulted multiple individuals that have built them, themselves, the in house capabilities. 

[00:12:10] And just based on the information we got from them about the cost of that, it’s just not something that I don’t think we’re going to do in the nearest future. 

[00:12:20] Perhaps, in maybe 10 years when we’re running really thousands of tests, that would make sense. But right now, I don’t see that benefit. 

[00:12:32] Vipul Bansal: Got it. Let’s move towards the most inspiring bit which most of the audience waits for, is the examples.

[00:12:41] I’m really curious to know, without sharing any sensitive or confidential information, if you could share with our audience, some interesting examples of experiments that you’ve ran at Tele2. That’d be really, really great. 

[00:12:57] Maria Luiza de Lange: Yes. 

[00:12:58] So I prepared a couple of examples so that we can go through them.

[00:13:02] One of the experiments that we’ve run recently and I’ll share my screen in a second. It was a standard UX practice, that I’m hoping it can resonate with many of you, where we added a copy under the hamburger menu and the search icon on the mobile device. So we have run, as I’ve mentioned, what’s on the mobile device.

[00:13:23] We added these wordings under the search menu and hamburger menu on mobile and we could see that the interaction with these elements has increased by 4% and we also had indication because it wasn’t statistically significant. But we had positive indication in terms of increase in transactions.

[00:13:46] So I think this is an element that most of the eCommerce websites have and if you yet don’t have these standard UX practices implemented, I think this is a nice quick test that gives a lot of positive impact. So that’s one test we had done. 

[00:14:05] The other one before I jump to it, is about how we work with consumer feedback as well.

[00:14:15] So as many of different organizations, we also have several tools that we use, from which we get data to frame our hypothesis. One of it is voice of consumer, which is our survey tool and there we have seen a lot of consumers were complaining that they just would like to understand what is the pricing of our broadband offering before they go through what we call a feasibility field. 

[00:14:38] So for us, whenever we offer a broadband subscriptions, we need to know consumers’ address to be able to check which network they are attached to, so to say. 

[00:14:51] So for us, it’s very important before we give consumer indication on pricing is that they fill in this feasibility. But still in the feedback from consumers, we knew if you look on the left hand side, we had the no indication of pricing and consumers don’t want to go through that barrier and the hassle of filling in feasibility if their pricing indication is any way out of their scope.

[00:15:17] So what we have done is, we added this from prices because again, depending on which network the consumer is, they might get different price but we gave consumer indication of these are the different speeds that we have available and this is the pricing you can expect more or less and we added the disclaimer that, of course, it depends on your address.

[00:15:36] But what we could see is that, of course, we saw a decrease in feasibility to start because those consumers that just wanted to see the price would not fill in the feasibility. But what we’ve seen is that, actually the whole conversion of the funnel, so from the people that started feasibility going through all the stages of the funnel and transacting has been positive.

[00:15:58] So we’ve had, almost 12% improvement in the conversion rate of that particular flow of the website and we’ve had 12% increase in transactions just by showing consumers indication of prices. So that has been interesting. 

[00:16:14] And the last test that I have prepared is connected to sustainability.

[00:16:18] So at Tele2, sustainability is really important for us and what we wanted to test is within our mobile subscription flow, before we were defaulting to a physical plastic SIM card and what we wanted to test, as we now introduced an online flow for activating of e-SIM card.

[00:16:40] We wanted to default to that which is a better sustainable solution because obviously we avoided the production of all the plastic and we added a copy of, why this is a better sustainable choice.

[00:16:51] How much plastic do we save by going with the solution. And we could see that the share of e-SIM within our subscriptions has increased. So you see the number of e-SIMs sold has really increased and also that has given positive impact on the transaction as well as like indication of the transactions.

[00:17:14] So those are the three kind of examples that I have prepared in terms of, sharing the results. 

[00:17:22] Vipul Bansal: Really insightful examples. Thanks for sharing these examples, Maria. 

[00:17:26] That makes me curious, your first example, about just adding the text search and menu below it could make such a difference, it’s something which I think many businesses would be skipping.

[00:17:43] So do you recall any other example, any such example where in it was a low hanging fruit, because we as experimentation professionals or experts, what I have observed is they usually run after very complex cases that they can make. Pulling in tons of data, running analysis on them and then moving on with the experiments.

[00:18:11] So are there any low hanging fruits? Any more that you could talk about? 

[00:18:18] Maria Luiza de Lange: Yes, absolutely. So just want to think from top of my head is an experiment we run recently with our streaming team. So obviously within the entertainment area, we have both the kind of standard linear TV as well as streaming packages and for some reason or another, we’re also in this kind of call based transformation project.

[00:18:40] One of the models within our CMS was not developed the same way that all the other product listing pages on the website. And basically that when you have different packages the whole tile was not clickable. And as we know especially on mobile, the consumers just tend to click on overall package CTA. So just by implementing that simple change, which is a UX standard practice to make sure that your whole package, regardless whether it’s product title, description or the CTA, that it’s all clickable, just by making that small change, we have improved consumer experience.

[00:19:16] So absolutely, low hanging fruits are a part of our journey and things that we run at Tele2. Yeah. 

[00:19:23] Vipul Bansal: Awesome and experimentation, it could be as simple as you just mentioned, finding the low hanging fruits in this manner or it could be as complex where you have to pull a lot of strings, right, to get an experiment to be executed. 

[00:19:41] Do you have any experience pulling all such strings together to basically execute a really complicated experimentation program or an experiment wherein you have to put a lot of effort and you are very convinced that, hey, this will work out.

[00:19:56] So, it will be great if you can share anything from that POV as well.

[00:20:01] Maria Luiza de Lange: Sure. So, we try to avoid very complex experiments that don’t bring back the value in terms of development time spent. So just to mention that. 

[00:20:13] But there are a couple of tests that were quite complex that we did run because we saw a big kind of commercial benefit of doing them.

[00:20:22] So obviously anything that has to do with relation to pricing, whenever you change anything on the pricing, that becomes immediately complex because you need to update several pages and make sure that it is consistent from product listing page, product detail page, your order summary, checkout, confirmation page, confirmation email and so on. 

[00:20:44] So that requires a lot of different stakeholders to be aligned and one example of the test that I can mention that we’re actually planning on launching soon, is just changing product name. 

[00:20:58] So similarly as to pricing that I just mentioned, changing product name can impact a lot of different touch points from product listing page to product detail page, to your order summary, to the product confirmation page, to the product confirmation email.

[00:21:12] So. It does require a lot of alignment, both with commercial stakeholders. I’m thinking here in terms of sales managers, as well as email managers, the people that are sending communication or the automatic emails that are being sent to consumers after purchase. 

[00:21:28] From content managers, in within CMS and of course the core experimentation team. So yes, there are a lot of complex tests. Sometimes it even involves, representatives from finance team. 

[00:21:43] When we start talking about a cost vs increase in sales scenario, so experimentation definitely is a lot about planning. I believe someone says it’s like “If you cannot plan with CRO, you won’t succeed”.

[00:21:59] So it’s a lot about planning, but it’s also a lot about stakeholder management and making sure that you involve the right people and you collaborate as much as possible and break these, silos that tend to happen in big organizations. 

[00:22:14] Vipul Bansal: So, are there any consequences to running experimentation as well?

[00:22:19] Maria Luiza de Lange: I mean, of course, I would say, I think the one that is very obvious is the performance impact.

[00:22:29] But again, that’s I think the choice that one makes and we are in very close contact with our developers to make sure that contact and the way we implemented our chosen vendor is as minimal as can be. So whether that is, server side or just making sure that impact for the consumer is as small as can be.

[00:22:52] Vipul Bansal: When there are consequences, when there are risks involved in running experimentation, of course, you do mention that, you make sure you speak to your developers and make sure that there’s no flickering and the page speed remains faster. What other risks are there if there are any?

[00:23:13] How do you manage those and how do you basically mitigate any such negative impacts on user experience and the overall business outcome? 

[00:23:21] Maria Luiza de Lange: So I do strongly believe that the whole purpose of running experiments is to mitigate the risk. So to make sure we avoid releasing something that can be potentially harmful to both sales, but also consumer experience, or that it doesn’t make any change.

[00:23:41] So, I believe within the CRO community, the Harvard Business Review study of 1000 experiments, is quite known where the outcome was that 90% of tests, either had a negative impact or didn’t add any value and that’s just we need to ensure that whatever changes and the decisions that we make that can have impact on the business.

[00:24:14] We need to make sure that we test basically, that’s what I think the best way to avoid the risk is basically to test because then you can have your guardrail metrics and make sure that whatever you release to production is not negatively impacting your sales or performance or user experience.

[00:24:35] Vipul Bansal: You know all this while, do you have a list of things that you believe that shouldn’t be tested on a website? 

[00:24:40] Maria Luiza de Lange: Tele2 encourages the business to test almost everything. I honestly cannot think of things that you should not test. It’s a very tricky question. 

[00:24:50] Perhaps in the future, there will be some things that I can get back to you on that one, but I mean, from whether that is even the API you want to do, or whether that is just a UX or UI or the feature you’re bringing to just releasing the feature and having guardrails metrics on newer sales.

[00:25:12] Obviously, as I work with CRO, I do think we should test the maximum of what we can and I think the challenge is more today that unfortunately still a lot of what we release is not tested. So I think the direction should be rather to testing more rather than thinking of what we shouldn’t test.

[00:25:33] Vipul Bansal: With regards to experimentation at Tele2, how our customers involved in this entire process. Do you speak to them for direct feedback? Or do you roll out surveys? How does it happen? 

[00:25:48] Maria Luiza de Lange: Yes. So we do review thoroughly, our voice of consumer feedback that we get from consumers. We treat it very seriously. And we had actually a case recently within one of our flows, it’s called family flow, where we have noticed that consumers were struggling and we have found the issue. 

[00:26:11] We managed to replicate it and we actually fixed it within two weeks. So, that’s just one example of how seriously we do work with customer feedback. 

[00:26:22] We really do want to make sure that whenever possible, of course, we are taking it on board and we do listen to our consumer’s needs.

[00:26:31] That is absolutely number one priority to us at Tele2 and as I said, we really want to make sure that the consumers also have a way of voicing their frustration and there’s a saying, “Love your frustrated customers”. I think they just make it so much easier for us within the CRO area.

[00:26:55] So yeah, absolutely. Love to all the frustrated people there, buying Tele2 subscriptions, just leave your feedback and we will work hard to make sure that it’s better next time. 

[00:27:07] Vipul Bansal: This is one of my last questions, when it comes to running experiments, of course, you have to roll out the ROI, you have to display the ROI to your higher ups, right?

[00:27:18] So, how do you align all your experiments to the needs and all the goals of the business? 

[00:27:25] Maria Luiza de Lange: Yes. So as I mentioned at the beginning, we sell several products on our websites. And when we prioritize the tasks, we ensure that the tasks that we run are within the flows that bring us most revenue. And also prioritization of the test is actually done by the product owner in alignment with the business. 

[00:27:50] So the business has a direct impact on which tests we’re running and in what priority. So that’s kind of how we solve that challenge. 

[00:28:00] Vipul Bansal: We have come to the end of this really insightful discussion, Maria.

[00:28:06] One of the really interesting questions that our audience requests me to ask all the speakers and for me as well is what books are you currently reading? 

[00:28:14] And if you are not a book person, and even if you’re a book person, it’d be great to get some web series recommendations from you as well.

[00:28:21] Maria Luiza de Lange: Sure. I do read books. I think we’re always more ambitious in terms of how many books we read. There’s a whole list of books that I usually read that once and also a TV series. 

[00:28:37] So some of the books that I currently read, so one for work is the ‘Trustworthy Online Controlled Experiment’ by Diane Tang and Ron Kohavi.

[00:28:48] For entertainment, I’m currently reading the book called ‘Spare’ by Prince Harry. It’s really insightful into the whole kind of royal life and how things are down there. And also with my friends, we have this book club where my current to do is the ‘Seven spiritual laws of success’ by Deepak Chopra.

[00:29:09] So those are the books that are currently on my roster.

[00:29:14] And in terms of TV series, the recent TV series that I’ve watched on Netflix was ‘Working: What We Do All Day’ where Barack Obama is also featured, and I really found this interesting. There was this quotes and obviously it goes through all the different kind of classes of society from the frontline workers to middle class to the 9% and 1% of society.

[00:29:37] So that was very insightful. But there was a saying that, ‘If you can afford everything that you need and half of your dreams, what you want, then you have a good life’. 

[00:29:47] And I liked that description, but it’s you know, that’s kind of saying a little bit because you still have some dreams that you’re unable to realize, right?

[00:29:56] So you have something to look forward to and aim for and of course you have this stable life so you don’t have to worry about paying your bill. So I just like that description of definition of what a good life is. 

[00:30:12] Vipul Bansal: Thank you so much, Maria. Can’t thank you enough for sharing such insights for our audience at ConvEx 2023 by VWO.

[00:30:20] Thank you so much for being a part of ConvEx ’23. 

[00:30:23] Maria Luiza de Lange: Yeah, of course. It’s been my pleasure. Thanks for inviting me.