Rapid Experimentation: How Testing Works at Scale

[00:00:00] Vipul Bansal: Hello, everyone. Welcome to ConvEx ’23, VWO’s annual virtual summit for growth and experimentation professionals. 

[00:00:14] My name is Vipul and I’m a senior marketing manager at VWO. Thousands of brands across the globe use VWO as their experimentation platform to run A/B tests on their websites, apps, and products.

[00:00:27] I’m excited to have Aditi with me here, who is a Senior Product Manager focused on digital growth at Thomson Reuters. 

[00:00:34] Hey Aditi, excited to speak with you. 

[00:00:36] Aditi Gupta: Hey Vipul, I’m glad to be here today. 

[00:00:40] Vipul Bansal: So let me start off with a quick introduction about you. A very basic question. So what inspired you to pursue a career as a product growth leader?

[00:00:52] Aditi Gupta: Interesting question. So I never set out to be in product growth. I in fact, spent my early years of my career in digital marketing, and somehow I was working on an app launch where I did the marketing. And when the app launched in the market, it did not do well. The marketing campaign was great but the app failed. 

[00:01:13] We had about more than 90% uninstalls in the first week. That’s how just working through figuring out how to take that product to market and to increase that, I would say, develop a better customer experience, I somehow transitioned from digital marketing into product and it’s been seven years and I’ve never looked back.

[00:01:35] Vipul Bansal: Awesome. So would you say that would be your most significant achievement or is there something else that you want to share with us? 

[00:01:43] Aditi Gupta: Yeah, in terms of significant achievements, there have been a couple. But something I can talk about is something at TR I’m currently proud of is that TR is a company which provides solutions for knowledge workers to increase their productivity.

[00:02:03] And one of the products that I work for, it has about more than 90,000 content pieces which are written by industry experts. And that content platform organically gets about half a million visits. 

[00:02:19] So, I worked with a cross functional team to utilize that platform to acquire customers. And that was one of the first at the company and also through that we developed use cases, which could be used in other verticals as well as other products. 

[00:02:39] Vipul Bansal: Also, I’m eager to know how does experimentation fit into this world of product growth? 

[00:02:45] If I would call it that way. 

[00:02:48] Aditi Gupta: Product growth is all about unlocking and figuring out hidden opportunities in your product as well as your business which you identify either with your team or it comes to market research and those opportunities are at times unknown.

[00:03:09] Maybe the company has not done them before. There could be some risk associated with them in terms of, will your customers like it or not? 

[00:03:18] Experimentation helps us validate those ideas. And how we do it? Experimentation is a process where you use low code and no code solutions to test an idea in a cheap fashion so that when you take that solution to market, you have some validated results that will work when it goes out in the market. And your customers will like it as well as the financial risk associated with it reduces.

[00:03:54] Vipul Bansal: I did anticipate that answer because experimentation, I think, is the right way to be wrong as I put it usually. And of course, it saves you a lot of cost, it saves you a lot of effort and I think it’s a most customer centric way to make any product decision, any growth decision.

[00:04:13] While I was researching more about you and figuring out what questions to ask you, I did come across your LinkedIn profile and I saw a really interesting concept there that you mentioned about rapid experimentation. 

[00:04:28] So, of course, I would like you to maybe share a few more tid-bits about what rapid experimentation is but also you mentioned that there’s change in processes when a business moves from SMB to an enterprise stage. So experimentation as a practice, how does it change from an SMB stage to an enterprise stage? 

[00:04:53] Aditi Gupta: A great question. So, before TR, I’ve always worked at SMBs and startups and I’ve been part of [SMB] teams. 

[00:05:02] SMBs are like smaller teams, you’ll probably have one product manager managing the entire website. So all you have to do is, you only have to take your manager’s approval. Get your team on board, your developers on board, your design team on board and you can literally set up an experiment in one day. And you don’t have to fight with other teams to run it or the buying process is much quicker and I would say much faster. And from my experience working through SMBs is the culture at SMBs and startups is break fast and learn fast.

[00:05:47] So it’s pretty, I’d say it’s easy to run and set up experiments in a SMB. But when it comes to an enterprise, it’s a completely different ballgame. 

[00:06:00] A checkout process on a website is managed by three to four teams. Every page on the website has a different product manager who owns it and they also have a different development team which works on it.

[00:06:15] So in terms of stakeholders, if you want to run an experiment on the website you’ll probably have to talk to maybe three to four people to even set up an experiment. 

[00:06:29] So that’s one of the main difference in terms of stakeholders. You have to talk to many more people.

[00:06:39] The second is in terms of execution time. Again, if you’re in a startup, you can literally set up an experiment in a day but when it comes to enterprise there’s a formal process which I’ve seen you have to inform everyone, you have to build alignment, and when everybody is on board then an experiment can go live.

[00:07:06] So the second is time to market, it takes much longer. But from my personal experience, how I succeeded in setting up experiments, at an enterprise level is that I come up with ideas, I talk to different teams individually or in a group. I present my ideas and also the experiment plan what I’m trying to do, what I’m trying to achieve, and also I always associate a financial benefit and a customer benefit with an experiment. That has helped me get buy in from stakeholders faster as well as build trust and social equity. 

[00:07:55] Vipul Bansal: Would I be wrong in opining that SMBs are not very much ROI focused compared to enterprise companies?

[00:08:05] Is that the reason enterprise companies are very slow in decision making. They appear slow in decision making because they have to be very careful about the ROI of whatever effort they put in vs SMBs, they’re like, hey, we just need to look at just one metric which is user acquisition. 

[00:08:24] Let me just get more and more people and I’m not concerned about the revenue and stuff. Is that true? 

[00:08:29] Aditi Gupta: It depends. I cannot give you a true and false answer because every SMB, I would say every company has different goals. So, I worked at both where a startup, their only goal, they were well funded and they were in a scale up phase where their only KPI was customer acquisition.

[00:08:52] Customer acquisition at any cost. So a lot of risks. The trade offs that they made were customer acquisition vs something else. But I’ve also worked at startups who are more conservative. They’re bootstrapped or they’re more conservative. They are still in that scale up phase but they’ve taken a more conservative ways. There, they invest in customer experience. 

[00:09:22] So there’s more, I would say thoughtfulness. When you run an experiment. Even before running an experiment, you do focus groups to really figure out what is the customer seeing? What are the likes and dislikes? 

[00:09:37] And based on that, you take a more conservative approach in terms of experimentation on what are the goals of the company. So your experimentation program should always be aligned to what is the short term and the long term goals of the company. 

[00:09:53] Vipul Bansal: In terms of convincing your leadership in an SMB company, the CEO might be involved in the experimentation program end to end. 

[00:10:04] In enterprise company, of course, the CEO cannot be involved but the senior leadership whoever is involved in the experimentation program, what have your observation been?

[00:10:15] Are they really involved in terms of knowing the nitty gritties of the experimentation or they’re just concerned about okay, so these are, this variation won, this variation didn’t win, what’s the recommendation? Let’s move on to the next one. What’s the scene like?

[00:10:31] Aditi Gupta: It really depends on the company culture and what part of the journey are you impacting? Is it a small change? Are you just changing the colors? Then I would say no leadership or nobody’s involved. 

[00:10:43] It’s literally your team who wants to make an impact. If it’s an incremental 1% – 2%, nobody cares. But literally if you’re changing the layout of your checkout page or you’re impacting something which will make or break the business. If you run an experiment and the checkout breaks or the checkout reduces by 30% then, of course, you have to take buy in.

[00:11:09] That’s a risk you have to take by in from leadership and they may not be involved again. All depends on what is the impact and if it’s something the leadership cares about or does it impact their top line metric. So again, like from my experience if there’s a high risk, I’ve always involved leadership so that I have buy-in and we can also turn off something quickly.

[00:11:38] So, give them that peace of mind that even if you’re experimenting this we can quickly turn off if we see our guardrail metrics getting hit. 

[00:11:51] But in an enterprise, the difference is there are different leaderships. The layers of leaderships is probably two to three times more than an SMB.

[00:12:05] So immediate leadership is involved in an experimentation program especially the big bets. Because in enterprise, the number of experiments that we run are lower but they’re definitely bigger bets. 

[00:12:19] Vipul Bansal: Also, you mentioned about culture and that is a great segway into my next question, which is around the most, I would say discussed topic around experimentation, building a culture of experimentation.

[00:12:33] And of course, there are myriads of other sources. And each source presents an insight of its own. So from you specifically because you have also worked on this as a project and also knowing that building a behavioral change in an enterprise company involving so many people is really really a tough task, right?

[00:12:52] So first of all, of course, what are your recommendations and advice? How do you bring a lot of people on the same page. And by same page, I mean everyone has to basically challenge their assumptions their biases whatever they’ve learned they have to basically unlearn that and let the people decide whether you’ve learned it correctly or not.

[00:13:13] So yeah, my first question would be, how do you bring everyone on the same page and build that culture of experimentation? 

[00:13:21] Aditi Gupta: Great question and again like, they’re literally eBooks written on this. I think one thing I’m really blessed with at TR is that the leadership is vested in the culture of experimentation. They’ve leaned into and they’re totally on board in terms of growing and developing this culture of experimentation. 

[00:13:43] So, that’s one of my recommendation that your top level leadership in any company, if they’re not involved or they’ve not leaned into a culture of experimentation no matter what you do as a product manager, you’ll never be able to succeed.

[00:13:59] So that’s number one, that you have to get your leadership team on board. But even if your leadership team is involved, that doesn’t matter that everybody on the ground and everybody throughout the ranks will be bought in. 

[00:14:15] It’s a little in terms of ways of working, experimentation teams are a little more out there, they’re a little destructive as compared to the traditional development process where you build requirements, you go to market. But in terms of experimentation you personally test, you see if a solution fits the market, if it’s meeting our financial or business goals. Then only you take it to market.

[00:14:42] So firstly, you have to build that, educate your peers, your cross functional teams who’ve never earlier had this concept or never worked with experimentation teams.

[00:14:56] A good way of doing this is when we started, we used to do bi-weekly shareouts. We would invite stakeholders, peers, cross functional teams involved in the experimentation process.

[00:15:13] We used to share experiments we are running. We used to co-create with them, invite ideas, ask people that this is an idea of an experimentation, would you like to partner with us? 

[00:15:27] So these things helped us co-creating our backlog of ideas we want to test. So that was one.

[00:15:35] The second was whenever I know we used to have bi-weekly shareouts, we used to share experiments, which we’ve won, which have led to some results, positive results and we also used to share failures. We experimented this, it did not work. So that is one thing that we owned as a team that we have to share both successes and failures.

[00:16:02] We also, in the same set up, also share our roadmap of what’s coming next. So, everybody our peers, stakeholders have an opportunity to lean in into what’s coming next and also give us feedback in terms of if something is a risky experiment or risky idea and then they help us guide through the approvals. 

[00:16:29] So I would say the pro tip is give them a heads up. It should not come as out of a sudden that one day they open the website and there’s an experiment running on it. Pro-tip, give them a heads up, get their buy in before running an experiment.

[00:16:47] Vipul Bansal: You mentioned about failures. At enterprise companies, the consequence of a failure could be like quite huge, right? A contrary perspective from you. Are there any risks to co-creating? Are there any risks to letting a lot of people run experiments? 

[00:17:09] Aditi Gupta: Yep, absolutely. I think personally I’ve always maintained that, your experiments should be aligned to your company’s goal and your organizational goals, your department, your team goals. 

[00:17:25] So if everything is stacked together then they won’t be conflicts. But yes, again if multiple teams are running experiments, they might be conflicts, like you are trying to improve the conversion rate from just the product descriptions to check out. But somebody on the product description page wants to increase a different method of payment, they would increase credit card transactions or payment through another gateway. 

[00:17:56] So at times, what they write on their checkout page vs what we write, it could be, our experiment would not give us accurate results because our experiment needs to go to checkout. 

[00:18:11] That’s a primary KPI. But they are trying to figure out a different payment method. So again, these things will happen but you have to make a process and the way that to do it ideally is you have different teams come together. 

[00:18:30] What we did in all earlier days, we came together and shared our experiment ideas that what we want to run and then again, we had an honest discussion that. Who’s going to go first? What is the criteria? And the way that we prioritize our experiments, who would go first, who would go second, so that the experiments won’t clash, is by looking at what are the goals for the team for this quarter and for the next quarter.

[00:18:59] So again, there is no one correct way. It’s literally as whoever is involved, whichever teams are running experiments have to come together to talk and then figure out the prioritization process. 

[00:19:20] Vipul Bansal: Right. So the key insight here is the visibility across the board, right?

[00:19:26] Everybody needs to know what’s happening, right? There has to be a center stage where in everybody has to assemble and have a look at okay, this is being run, this is not being run and whatever the programs are, and then take a call. 

[00:19:38] Aditi Gupta: Yeah. 

[00:19:40] Vipul Bansal: Yeah. Great insight there. 

[00:19:41] You also mentioned earlier about product roadmaps.

[00:19:47] And of course, as a product manager, you would be creating a lot of product roadmaps as well. 

[00:19:53] I’m keen to know how experimentation can inform and also sort of influence product roadmap. 

[00:20:02] Aditi Gupta: Okay. So this is one of my favorite questions. 

[00:20:06] For me, a product roadmap is a series of bets that you place on your product, which will eventually lead to customer impact or business impact and the only way you can de-risk this roadmap is by experimentation. 

[00:20:26] By experimentation, you create validated solutions. You go into the market at early stage. You figure out which features or which experiences work with customers and then once you see the business benefit of those, then you take those features and put it in your roadmap because eventually when those go to market the business results would be 2x or 3x than the current. 

[00:20:58] Experimentation is the only way or I would say one of the ways you can de-risk your product roadmap.

[00:21:05] Vipul Bansal: As most people ask what metrics do you look at when assessing the success of the experiment? So, I would also ask the same question to you. 

[00:21:15] Aditi Gupta: It always depends. Again, like I said, what is the goal of the company? And what is the goal of your team?

[00:21:22] Currently I manage customer acquisition and trial conversions. So the metrics that I’m married to is number one, the visit to conversion percentage and the second is trial sign-up to conversion percentage. So all the experiments that I do is in service to these two metrics.

[00:21:47] Vipul Bansal: So, curious to know. 

[00:21:48] Based on analysis of these metrics, how do you know whether to scale or whether to pivot in your experimentation journey? 

[00:21:57] Aditi Gupta: So in terms of analysis, if it’s a good fit to take it to production, we look at two things. 

[00:22:05] Is it impacting our metrics? That is the conversion rate.

[00:22:10] Ideally we look at a 30% plus conversion rate to make sure that it’s a success or not. And the second thing that we do is a cost benefit analysis. 

[00:22:24] What that is, how much money or what is the cost would the solution need once it’s deployed?

[00:22:33] Because in the enterprise world the cost of building is extremely high.

[00:22:38] So we also look at cost and then we look at how much financial benefit it will lead to and we do a cost benefit analysis. So once we know how much in terms of financial impact that we’ll have, a cost it will have, then we make a decision or a business case that should it go into our roadmap or not?

[00:23:01] Vipul Bansal: How do you ensure that the experimentation efforts are aligned with the overall business strategy and goals of TR? 

[00:23:09] Aditi Gupta: I think TR does a really good job of publishing its internal company goals. And based on those company goals, each department, each team, and even I at individual level, have to project our goals for the year.

[00:23:25] So everything I do is all in service to the overall goals. That is my team goals, my department goals, and the company goals. 

[00:23:33] So they’re all stacked together. So I think that’s one thing which really helps me align my team as well to a goal which is focused on the individual which will impact their individual performance as well as the company’s performance.

[00:23:53] So I would say that’s one pro tip I have that always have your experiment goals stacked and aligned to your company’s goals, your company’s short term and long term vision. 

[00:24:06] Vipul Bansal: I imagine in this entire journey of running experimentation, you must have faced a lot of challenges, right?

[00:24:12] So of course, I would be curious to know what challenges have you faced and how have you overcome them? 

[00:24:19] Aditi Gupta: So again, experimentation is a little different from the regular ways of working and challenges, I also also view them as opportunities. But one thread that I’ve learned throughout my product growth career is that you have to get people involved.

[00:24:39] You have to get cross function teams involved. You have to get your leaders involved and have everything on pen and paper. Even when you talk to your stakeholders always have a written document which talks about the customer research that you’re seeing or the data that you’re seeing.

[00:25:01] Let’s say what your experiment is in service to, what are the primary metrics, what are you trying to derive out of it?

[00:25:11] Are you trying to make a customer impact or a business impact clearly write down your entire thought process that helps you take your stakeholders and peers, whoever you’re influencing through your thought process and also a journey into that entire experiment. So it should be well thought out.

[00:25:33] Vipul Bansal: What do you see as the future of experimentation in product growth? 

[00:25:37] Aditi Gupta: I think experimentation is here to stay.

[00:25:40] In the past, experimentation has always been like a part of special projects and outside the traditional product development framework but as we move ahead, I see experimentation and product management going hand in hand.

[00:25:56] I see them getting more integrated into the day to day workings of a product team. 

[00:26:07] That is one and with tightening of budgets as well as the high cost of tech talent throughout the world, the cost of building solutions and taking them to market is getting higher and higher each day.

[00:26:23] And one of the ways that you can de-risk or reduce that cost is experimentation because experimentation helps you validate solutions in the pre-launch phase.

[00:26:34] You can prioritize which features and what products are going to market based on the results that you get from experimentation.

[00:26:44] Vipul Bansal: Great, thanks for sharing your observations and your knowledge with us. That brings us to the end of this discussion and my last question as always, I think the most requested one I could assume from our audience members and it’s around the books that you’re currently reading. 

[00:26:59] So definitely would love to hear from you, your recommendations on what books are currently reading. And also if you’re not a book person, could you also share the names of some web series that you’re currently watching? 

[00:27:11] Aditi Gupta: Oh, I’m totally a book person. Let me show you a few books I’m currently reading..

[00:27:16] If you can see these are the three. And yeah most of them, like two of them and since we’re talking about the concept of experimentation, I would recommend everybody to read this book ‘Experimentation Works’ as well as ‘Disruption Proof’, these two books talk about how you can navigate enterprise organizations and build a culture of experimentation. And as you grow in your career how do you get teams and leaders bought in to the idea of experimentation and business growth. 

[00:27:59] Vipul Bansal: One of the book that is on my wishlist for a time now is ‘Experimentation Works’ by Stephen Thomke, I remember it was released I think back in 2019 and since then it has been lying in my wishlist but I could only afford to buy these books.

[00:28:18] By the way, the top stack is still half read. So I’m guilty of not reading a lot of books, so I’m not buying any new as well. 

[00:28:25] Aditi Gupta: Oh, it’s the same with me. I have a stack of probably these many books which are half read and like unread.

[00:28:32] I’m a big geek for business books so yeah, it’s the same with me. 

[00:28:38] Vipul Bansal: So, a virtual high five on that. 

[00:28:40] Aditi Gupta: Same here. 

[00:28:44] Vipul Bansal: Awesome. Awesome. Awesome is all I can say as we bring this discussion to a close, it has been a real pleasure talking to you, Aditi. 

[00:28:50] Thank you so much for taking out your time today and choosing to speak with us, sharing your insights and observations. I’m sure the audience would really love this session and must be scribbling their diaries with all the insights that you’ve shared.

[00:29:04] So thank you so much for that.

[00:29:10] Aditi Gupta: Thanks for having me.