Data Dilemma

Hi, everyone. Thank you for joining in. I’m Siddharth from VWO. I take care of marketing here. At VWO, we do a lot of knowledge sharing sessions for our audience so that every conversation they have at least one takeaway so that they can implement it in their marketing careers.

Today we have Laura who’s joined in, and she’s gonna talk about something which is super common in our day to day work, data dilemma. Various situations in which we are confused or on how to go about, taking decisions or the next steps based on data or the lack of it. This is going to be about a 45 minute to 1 hour conversation. You can ask questions at any point in time. There will also be a dedicated Q&A section in the end. You can ask questions on chat either directly to the organizers or, public global broadcast, and we will also have a few polls through the entire conversation.

Do answer so that we can get real responses from people who are actually practicing marketing or some field that really depends on data. With that, I would like to hand over the stage to Laura now. Laura, please take it forward from here. Thank you so much.

Laura Amin: Thanks, Sid. Hi, everyone. Thank you so much for joining. Just a little bit of an introduction about myself. My name is Laura.

I am currently in Toronto, Canada, and I work for ecobee, which is a smart home company. I used to be in Bangladesh before, which is where I’m originally from. And, yeah, so I’ve been in marketing throughout my entire life. I have dabbled in product management, product marketing, operation, strategy, the whole thing. So, hopefully, I will be able to share some insights that are helpful for you and you get to take away something from here.

And if you have any comments, any feedback, anything to add, please do so on the chat box. I’m sure there’s a lot that we can learn from each other. Okay. Let’s get started. Is my screen visible?

The slides are good. Sid?

S: Yes. All good, Laura.

LA: Okay. Awesome. Thank you. Alright. So let’s get started.

The first thing I just wanted to share is what I’m gonna be going through today. So I’ve broken it down into 4 sections. The first part is just focusing on what are the different types of data that exist. The second part is taking that data and just looking at what sort of data is relevant in different product life cycles. The third part is really focusing on what you do when you don’t even have the data.

And then finally, when you’re working with data, what are the different kinds of traps you can fall into and how do you avoid them? And, hopefully, at the end, I’ll also just wrap it up with a few takeaways so that you can remember at least one out of those 3 and it helps you in your regular work. Okay. So just in general, I think there’s, like, really broad 2 groups of data. There’s qualitative data and there’s quantitative data.

So qualitative data is mostly for, like, open-ended questions where you’re getting a bit of description. So it’s not necessarily a numbers based thing. It’s very much open-description. The kind of research that’s done is in-depth interviews or focus groups or even, like, say, open-ended questions inside surveys. So that’s where you get qualitative data from.

And then, of course, quantitative is when you have a number aspect to it. So generally speaking, it would be close-ended questions sent to a big group of customers so that you can get scale and you can get statistical significance. And that’s how it comes together. So one is less numbers oriented, more description oriented, and the other one is numbers oriented. So the first poll that I wanted to do is just ask the audience which type of data do you think you should go after first?

Sid, do you mind pulling up the poll for this? I’m not sure if it’s working because this is my… Oh, there you go! It’s working. Hopefully everybody can see the quick poll. The question on it is which type of data should go first.

And then the options are qualitative or quantitative. Sid, can you tell me what the results are? Oh, interesting. 100% quantitative? Wow.

So generally the way I think about it is it really depends on the scenario. And I say that because, for example, say you’re going after a new product launch in the market. So when you’re trying to do that, really, you’re trying to learn about what the customer is thinking, what is the kind problem they’re having. What are their jobs to be done? And so when you’re going after that, it’s hard to do it quantitatively.

You really have to ask open-ended questions and try to get a sense of where the customer’s head is at. However, if you have some known data pieces already in place. So maybe you have some secondary data available. Somebody else has done the research, or it’s a product that’s already in the market and you wanna know a little bit more about it. At that point, quantitative would be relevant.

But in my mind, almost always ‘quali’ goes first when there’s more ‘unknowns’. And then when there’s more ‘knowns’ and you know a little bit more, that’s when ‘quanti’ comes into play. A lot of the times when you do a ‘quanti’, it can get to the point where you’re like – Oh, yeah, 75% of customers said yes to this type of a product. But you don’t necessarily know why they said yes to that type of product unless you’ve really dug deep and understood what sort of problem or what sort of need was really driving that. And so you might fall into the trap of taking away an insight, which was not really true.

And I’ll talk about that a little bit more as I go along. But I think we should always know that there’s merits to both, and you should kind of use both whenever you can. But don’t do them simultaneously. Do them sequentially. And then depending on your scenario, it should be which one goes first.

But it’s interesting that everybody thought of quantitative first. Okay. So this one is just really understanding what the levels of data can be even within the qualitative and qualitative. So the first portion is, of course, the business line. And I am assuming that at any point of time in your professional life, you’re responsible for either a product, a business line, a service, whatever it is.

And so the business line is sort of like the cusp of it all. That’s the thing that you should definitely know about, mostly inside out, generally speaking. The second level is company. So on a company level, what you’re trying to understand is where does this business line fit? Where does it sit?

So for example, if your company only has a single product, then obviously the company and the business line kind of becomes synonymous. But a lot of the time, you will have different product lines. So for example, if you work in telecom, you’ll have a data line. You’ll have a voice line. You will have some AI lines as well. So even with ecobee we have multiple lines.

We have the thermostat. We have our home security solutions. We have some complimentary services. So there’s multiple business lines within the company. It’s important to understand where you sit there and what sort of contributions are expected out of you.

So that’s why a company becomes important. For example, say we’re talking about the budget available to you for your business line in a certain year. If it is a single business line company, you likely have the whole budget, But if it’s multiple, then you have to understand what sort of budget is available to you, what sort of investment is available to you, what sort of support, what sort of resources you can get. So that’s the 2nd level of understanding. The 3rd level is competitors. And so this is super duper important, of course, because if you’re bringing something into the market, and you’re not differentiated or you’re not desirable to the customer compared to the competitors, then, of course, you’re not gonna get the traction that you’re looking for.

The things that you kind of need to know over here are not just for the competitors now. You also need to have a sort of a future view. So for example, the competitor is bringing in NextGen Technology before you. So, when we were in tel, I used to work in telecom before there’s a few telecom examples there. So back home when we were launching 4G, 5G, it really depended on which operator brought it first because they could claim it and they could be ahead of the game and be, like, the first network to bring 4G to your doorstep. The first network brings 5G to your doorstep.

Even over here, right now, I’m not sure if you’re aware of it, but there’s this standard call matter that everybody is speaking of. And so it really depends who brings that first to the market, and then they can claim that they have created that differentiation. So you have to also have a pulse of where your competitor is really headed more than even more than what they’re doing right now. Because you wanna be at least a couple of steps ahead of them if you really, really, really wanna be ahead of the game and get that traction and that customer love. 

The 4th level is market. So the market is really understanding overall trends, not just competitor specific trends. So when you think about the market, it expands beyond your immediate competitors. So just as an example, say for example, the gas prices right now are super high in North America. And so what this is doing is impacting Uber and Lyft surge prices, which were already pretty bad to start with, and they’ve gotten really worse right now. So if you were thinking about the ride sharing industry and you were at the competitor level, you would just be looking at Uber, Lyft, Touro, and bike-sharing companies.

But when you start looking at the market, you start looking at beyond the competitors. So for example, this gas prices, which is almost like an 8G something, but it impacts the competitor. So you have to have that whole, like, level covered, and so that’s why the market becomes important. Now after all of this, to me, the absolute most important, even though I’ve called it L5 on the slide, I really think it’s almost like L 0. It’s like everything that envelops it.

It’s the context. So for example, in the whole gas prices situation, it really depends who you ask. So if your respondents, one of them is an Uber driver, which is their, like, that is their day job. That is what they do. And then on the other hand, you have a remote worker who is working at home and they don’t really need to travel as much. They live in a place where they have access to all of the amenities super close by.

And if you ask these two people about the impact of gas prices on their lives, you’re going to get really, really different responses. And so understanding that context is what is going to overrule every single data point that you have. In fact, even, like, within your company, if you’re having a conversation with somebody and you’re trying to get a data point out of them, It really depends what kind of day they’re having. Did they wake up feeling good? Did they wake up feeling bad?

They might not be as inclined to give you the data. So at any point in time, you always have to know or recognize that whatever data you’re seeing, it’s being filtered through somebody, someone, some trends’ context, so you should always be aware of that when you’re looking at any data. I’ve been speaking for a long time, and I wanna pause and see if there’s any questions. 

I must be doing a really good job explaining because there’s no questions. Perfect.That works for me. Okay. Alright. So going into the next slide. So this is like a very general product life cycle graph.

So the first portion is the new product development, which is when you’re just figuring out what to build, how to build it, and you’re building it as well. The 2nd phase is the introductions. This is when you’ve just launched it and you’re starting to gain momentum. The 3rd phase is growth, which is basically the biggest momentum phase. So this is when you’ve kind of crossed the chasm and so you’ve started growing into the market.

The 4th stage is maturity, which is at this point, your product is doing well. It’s almost like it’s a toddler right now. You don’t really need to take care of it anymore. It’s not a baby. Decline is, hopefully, this doesn’t happen, but sometimes products decline, sometimes their lifetime ends.

And so you have to let it move on to its afterlife. And this is like a very general product life cycle map that you would have seen in a lot of other places. Now the data points or the matrix really vary by which stage you’re in, and you should always be aware that different things are relevant at different points. And this is important because you don’t wanna be looking at tons of data at every single phase. Everything is not important in every phase.

And so if you try to cover everything, you’re actually gonna get lost and get very mixed signals. So it’s easier for you to kind of break it down by the phases and then just focus on what is relevant in that particular phase. Okay. If I can find my cursor and I can make this animation work, bear with me. There you go it works.

Okay. So for the new product development phase, I think of the top 3 things, and, of course, you can argue that there’s a lot of other matrices that can go in here. I’m just focusing on the top 3. So the first thing is Jobs-to-be-Done. Now what is Jobs-to-be-Done?

If you haven’t heard of this theory before, what it does is it tries to take a very deep look at what the customer needs instead of just doing it as a simple question. So let me give you an example. Say, for example, you need to drill a nail into your wall so that you can hang a picture. If you ask anybody what does this customer need, the likely answer is going to be a hammer. But that kind of takes it away from the actual job to be done.

The real job to be done is figuring out a way to put that nail into the wall. You can do that with a hammer. You can do that with a heavy book. If you’re strong enough, you could probably do it by your own hands.

I’m not recommending it, but just saying that it is possible. So you can do that. The whole Jobs-to-be-Done perspective takes you away from just thinking there’s a single solution to the problem at hand. So you’re drilling the problem down to its lowest smallest level so that you can understand what sort of different solutions you can bring in. What this also helps in doing is differentiating yourself because you’re not going at it with, oh, I’m just gonna build yet another hammer for this customer.

You’ll think maybe I’m gonna create a dual purpose book which tells you how to hang pictures and it’s heavy enough that you can actually use it as a hammer. Not suggesting that that is a definite product, but just for an example. But Jobs-to-be-Done really just boils it all down into a single problem statement, which is the second thing over here – It’s a problem identification.

Once you know the Jobs-to-be-Done, you can boil it down to a problem statement, and you can be like, this is the core problem that the customer has that I’m trying to solve for. And once you do that, you have what we call a product market fit, which is a very common term in product marketing. In marketing, sometimes in product management as well, but essentially what it means is you’re actually building something there is a need for. If you build something that the customers don’t really need or have other good options, then obviously by default, you’re not gonna get the traction you need. So Jobs-to-be-Done, problem identification.

And then the third piece, this is more of a business aspect to it, is the market size. If you’re creating such a niche product that it only works for five people out of a population of 500, you’re gonna have a hard time, of course, because you have to convince all 5 of them. You have to get a 100% of the market to go for you for you to even have a little bit of numbers on your business. So always try and see if you’re able to get to a market, which has a larger population so that even if you can penetrate a smaller size of that market, you’re still doing well and your business has potential to grow. So those would be like my top 3 in terms of new product development matrix.

And I lost my cursor again. Bear with me. This is a new tool that I’m using. I mean, it’s cool. Thanks, Sid, but it’s taking a while to get used to.

Okay. Alright. So during the introduction phase, there’s a system matrix, output matrix, and input matrix. So a system matrix, this is called differently in different places. I’m using the most generic term over here.

I’m sure my product friends will have a very nice fancy term for it, which I’m not using right now, but what I really mean by system matrix… 

I’m sorry. I guess there’s something in the chat, which I wasn’t really following. My bad. Okay. So for a system matrix, basically, it’s figuring out, does the technology work?

Say, for example, you’re selling a physical product and you have an add-to-cart mechanism, can customers click add-to-cart? Does the cart work? Can they check out? Do they have any technical issues? Can they complete the funnel and so on?

So it’s essentially just a technological check of does your product work or not. Then we have an output matrix. And this is something I actually learned super recently. So the output matrix is the single most important metric for your business. So what this really means is say, for example, revenue, that is your major thing that you wanna go after in that business.

Say, for example, it could also be sales units, like your main core north star, is to be able to sell a certain number of units. So that’s your output matrix. Your input matrix is what goes into that output matrix. So for example, if you’re going by the revenue example, revenue can be broken down to the number of users and then the price that the user is paying you. So that’s the level of the input matrix. Now why do you need to know your output matrix and input matrix?

It’s because you need to be able to judge the health of what you’ve just launched. If you can’t figure out if you’re doing well, how would you know if to improve, hold steady or phase it out earlier, then it reaches even before it reaches majority. So it’s super important to be able to understand or know for a fact even before you introduce, what is your biggest north star metric? And then what are the different matrices that roll up into it? So try and define those early on.

Okay. And then for the growth phase. Now this is where you’re having rapid growth, you’re acquiring a lot. But if you have a leaky bucket problem where you’re like, you’re pouring water into the bucket, but it’s got holes in it. So you’re also leaking the water out. And if you have more holes than you’re bringing in at the top, then you’re not gonna be left with anything.

So it becomes important when you start growing into that rapid growth phase that you have an eye on churn. And so churn in my opinion, hence is the most important metric to look at during the growth phase. You can obviously have, like, benchmarks where you’ll be like, okay, if churn is within 10%, I’m not going to work on it. As long as your net growth, which is basically the growth minus the churn, is good, that’s fine.

But a lot of the time churn creeps up when you’re not looking at it because you assume and this is not you, like, pointed at you. But a lot of the time, product marketers, marketers, product managers, we are in love with what we’ve created, what we’ve put out there, and so we think there’s no issues. And then they suddenly creep up, and they start creating the whole leaky bucket situation. So it becomes difficult to manage once it’s so much higher than you want it to be or what you need it to be. And so it’s better if you start looking at it early on in the growth phase and then always keep an eye on a no for yourself.

What is that benchmark or what is that number after which you’re gonna go into full on panic mode and fix this? Before anything else. So that’s the growth phase. Just in terms of the SaaS service, So SaaS is basically Software as a Service. It really depends on how you have your pricing structure created.

So, for example, if you have a monthly pricing, then your likelihood of churn is happening literally every month. And so that’s when you have to keep an eye on even more closely than if you had, say, an annual pricing structure. Because at that point of time, you have to wait the whole year to know if that customer is churning or not. Of course, you will have drop offs even before that. But because you’ve already been paid for the whole year, you might not care as much, which you should.

But even in that situation, depending on the kind of product you have, your churn might vary. And you can also use, like, industry comparables. Look at people who are already out there who are doing it, who might be having similar or adjacent products and what their churn looks like and benchmark yourself against that. Okay. And then on the maturity stage, really this is where profitability starts becoming important.

So till now, a lot of the time people will just focus on the top line, be like – Okay, gross margin is all that I need. But then over time, when you’re maturing, you need to be able to cover for the cost that you’ve incurred, not just variable, but the fixed cost as well and reach a breakeven point. So at this time, profitability is what becomes important. Then you can also kind of break it down.

So you can say that I’m gonna be profitable on all direct expenses first. Then you can say, okay, after I’m profitable in direct expenses, then I’ll start contributing to the overhead of the company. And then finally, on the EBITDA level, I’m gonna be profitable as well, or I’m gonna break even. So fix those benchmarks or those yardsticks in advance, and then you can work towards it. If you start thinking about it at this stage, and start measuring at that stage, then it’s gonna be a problem because you might not know exactly how to measure it.

You might not be able to figure out how to figure out the allocation of the expenses and so on. So I guess what I’m trying to say is by default, even before you hit that phase, in the pre-phase, you should have already figured out how to measure it and what is your yardstick for that measurement. So maybe you don’t look at it as often, but you have it planned out so that when you do start looking at it, you know instantly if you’re doing well or poorly. And then finally… Oops.

Okay. So if it’s in a decline stage, that’s when you start looking at what is salvageable. And what I mean by that is, for example, say I run a subscription service right now. For some reason, it’s declining. I’m going to phase it out.

That product will not be in the market anymore. At that point in time, I wanna look at what I have already built up. So for example, the whole subscription process, the subscription system, the platform, like getting those monthly payments, starting a trial-enabling cancellations, pauses, all of that capability is already built. So when I’m really thinking about what to do next and what is salvageable, I’m gonna try and use this capacity or the capability that my company and my product line already has to jump off into the next thing. So whenever a product is in decline, don’t think that it’s all going to go away.

Try and figure out what you can use from what you already have to solve the next big problem. And what this does is it brings down your operating expenses and it also brings down your investment in the beginning. Right? So you get a higher or a longer runway to get to the point where you need to with your product line. 

Okay, now, the fun part. I think we have a poll for you, Sid. And there you go. The poll’s up. Just wanna check-in on this one.

Have you ever had a situation where you did not have enough data or has your life been perfectly dandy and garden-like? Ok. Waiting. Of course. Yes.

Thank you. I am so glad to know that everybody has had disturbing situations like mine. I think it’s just natural. Like, you can’t always expect to have all the data, honestly.

Like, that’s never gonna happen. A lot of the time you will not have it and you’ll have to figure out how you work around it. So my suggestions. I think from the introduction phase onwards, if you remember the graph, actually, maybe I’ll hop over here. So you’d remember it.

So this is the introduction phase, the 2nd phase. And then from the introduction phase onward, have one suggestion. And then for the new product development phase, I have another suggestion. So from the introduction phase, like I was saying, you should have it have it. You should know what your output metrics are.

You should know what your input metrics are. And you should not launch if you don’t have a way to measure them. We’ve had this situation happen a million times. I’ve had this happen to myself as well where we were super excited to get the launch going.

We launched it. And then we were like, oh, crap. We don’t even know how to measure what’s happening. And so when we start looking at data points, we’re like, oh, we wanna know what the funnel conversion looks like. And it turns out we never really put any tags or any events or anything.

So we can’t really measure the funnel conversion. So if when it’s low, we’re like, we don’t know what’s going wrong. But because we hadn’t planned in advance, because we hadn’t put everything in place to be able to measure it and advance, we were stuck. And I would definitely recommend that before any introduction, one of the things you should check when you do a trial, when you do a customer trial, or when you do an employee trial, make sure that you check your data. Make sure you have a plan for how you’re going to measure your output matrix, your input matrix, what they are. And then check during that trial if you are even getting that data properly or not because you might have a solid plan.

Like, okay, this is what I’m gonna measure. I’m going to measure each step of the funnel, and I’m gonna see what the page views look like, what the unique page views look like, I’m going to be able to measure my add-to-cart trade. I’m going to be able to measure my final checkouts. But then if you haven’t really checked that data is flowing in correctly, then it’s gonna be a problem regardless. So plan it out in advance and try and not launch before you have it because the minute a product line is launched, you end up having to answer.

Are you doing well? Are you hitting your targets? A lot of the time targets are just units and revenue, but if you have a target of 100 units and you’re hitting 10 units, and you don’t have a view of how it all comes together and why it’s 10 units and not 50, not 60, not a 100, how would you even answer that question? It puts a lot of pressure on the marketer, the product manager, whoever is owning that business, And so try and make sure that that data is available.

If you don’t have it, don’t launch it. That’s a hard line. I know. I know, but it’s just we’ve been burned by it quite a few times. And so at this point, I’m very much like, if that’s not a checklist item we’ve checked off, we should not launch.

Delete the launch if you have to. Nobody’s gonna die. Hopefully, none of your doctors, I’m hoping. If you’re a doctor, do it. People are gonna die.

Okay. So the development phase. So when you don’t have the data, what you do is you model it out. So you model it out on paper, you model it out in reality, and then you validate. So, essentially, what you’re doing over here is ROI, which is Return on Investment.

So this is where you create a business case. You figure out, okay, what does my number of units need to be? What does my price need to be? What is the eventual revenue that I’m going to get? What does that mean in terms of the expenses that I’ve made or that I will have to make if I wanna launch this business, if I wanna launch this product?

And then you do sensitivity. And so sensitivity modeling is essentially figuring off which are your key levers of your business. So by default, you can say it’s the revenue and the cost at a very broad level. And then you push and pull and see what happens. So for example, what happens when I push revenue up 15%, that’s the optimistic view. What happens if my cost goes up 15% – pessimistic view.

But by doing that sensitivity modeling and trying to figure out the range you’ll see where what is an acceptable portion for you in terms of numbers? So that’s the ROI that you do. The second level that you do is figure out what the Minimum Viable Product is. A lot of the time this is also called Minimum Lovable Product. Different terms run around.

And for the minimum viable product for the MVP, what you wanna do is figure out the time required to build it and the effort required to build it. I do wanna pause here and do a poll around an MVP for something. I’m fairly sure we all use Netflix. So do you mind pulling that up? Thank you.

So this is just, like, 2 options in terms of what you think would go into the MVP for Netflix. Of course, you see a fully fleshed out version of Netflix now. This is just to see what your opinions are. If you were the PM of Netflix, what would you have chosen to put into the MVP? Oh, I love how the numbers are moving around.

I’m not sure if you guys can see the poll in progress but I can see how the numbers kind of move around a little bit. But I think we’re reaching consensus now. Nice. 30% of people think it’s intelligent recommendations and 70% think it’s information blurb and trailer for each movie. My opinion is that it’s an information blurb.

And I say that because intelligent recommendations are a delight factor. And so that’s something you create afterwards. Once you know for a fact that your product is going to work, that’s when you start building these, like, hooks or these things which create stickiness. So for example, I used to work on a subscription service business even before this one. So this is my second subscription service business that I’m working on.

So when you’re trying to do an MVP for a subscription business, I would be thinking about a few things. So for example, do I need to have a complete integration with a payment system? For my MVP, or is there any, do I need to be able to take payment from customers or not? The answer is likely yes for that MVP.

However, say I have a supplier who’s going to deliver the products to the customers, the end customers because I’m not building the product myself. I’m just the via. In that case, do I need to build that integration with the supplier for the MVP? I would say no. You can do a manual integration.

You’re obviously gonna have very few people in the MVP or, like, exposed to the MVP in terms of customers. So we can just do a manual integration where we, at the end of the day, create an Excel summary, you send it to them and they send it out. All we need to know is that the product is sent out. It doesn’t necessarily mean that we have to have a fully fleshed out back-end. So with the MVP, you try to create the smallest limited version of your product, which customers will still understand what you’re trying to do. 

So for the Netflix example, really what you’re trying to do is give them enough options for different kinds of movies, documentaries, and stuff like that, and they have to pay just a single price for it. So the intelligence recommendations I said is not part of the MVP because it’s not really the core of Netflix. The core of Netflix is you pay a single price and you get access to a million shit. Like, that library is huge.

So having that view of the library or giving that impression that they have a huge library is more important for their MVP. Then being like, hey, I can also learn from what you like and then recommend stuff that you would potentially like when you’re trying to look for other options. The first version of Netflix was really to be able to push that whole thing around, hey, we have a huge library, it comes at a single price.

So understand that whole, like, MVP situation. So you have the model on paper. You have the MVP, which is like a very small version of the product. And then the last thing you do is you do an experiment. So if you have any questions around, okay, what happens when I put the price at $10 versus $15?

Netflix, for example, instead of going ahead and doing it for a single price point, you go into an experiment. So you put it out for, say, a thousand customers each 1000 customers get $10,000 customers get $15. Everything else is the same, and you see what happens. And so that’s what you can do when you don’t have data and you only have guesses around what might work or not. So you’ve modeled it out on paper.

You know what the numbers need to be. You’ve created the MVP, which is the smallest version of the product. But it still shows the benefit of the product and you put it out there in multiple variations to a small group of customers and you see how they react. And honestly, a lot of the times, what data you get in surveys and research and everything is nothing compared to the kind of insights you get when you do this. Obviously, it means that your company has to have the capacity, the resources, the budget to be able to put this together because even an MVP is going to take resources that there’ll be a cost associated with.

The experiment is going to take resources. There’ll be costs associated. So there’s a limit to how much you can go. But if you have the option, if you have the liberty to be able to burn a little bit of money trying to test up an actual concept, always do that because it’s going to give you so much more insights than if you would have done a simple quantitative survey. Any questions on here, you can put them in the chat if you like.

Again, doing a really great job or people are asleep. Works for me. Okay. Before I go into pitfalls, which is basically where things might go wrong. I wanna do… I’m guessing this is the last poll.

I think this is the last poll. Yep. It’s the last poll. So do you mind pulling up the last one? Thank you.

So this is just an opinion. What do you think working with data means? Is it an art or is it a science? Ah, those numbers. Please vote if you haven’t yet.

This is a fun one that I always like to hear. Oh. Oh, wow. I think we may have 1 or 2 more people left to vote. Okay. And we’re closed. So we ended up with 44 percent art, and 56 percent science. I honestly did cheat. There should have been an option with both, and I’m fairly sure if I put that option in, a lot of people would choose that. But it was forcing a choice just to see where people end up.

It was interesting because when the poll first started, it was art-heavy. And then it started getting normal and then, equal and then science became heavier. So very interesting to see that. Anyway, the reason I was asking, data is not really in sight at the end of the day. And I say that because data is giving you a number.

So the number might be 5. It might be 10. You might have a benchmark for it, which is say 20. Is 5 good? Is 15 good? Is 18 good?

Why is it 5? Why is it 15? Why is it 18? You don’t know for sure. For example, say, the number of sign-ups you’re having in a period is slow, but the traffic coming to your page is the same.

Your in-funnel con conversion is the same. Your price is the same. So, what the heck happened? You’re seeing a data point and there’s no explanation or insight as to what happened. So that’s why I say a lot of the time the only way to avoid taking data for insight is to understand that you kind of have to combine art and science.

So you have the science of all data available and you know how the data works together and comes into that final output matrix. Which in this case, in my example, is a sign up, and then the input matrix for traffic to your page, conversion on your funnel, and the price. But you don’t know for sure exactly what’s happening. If everything is right and you’re still not getting what you need to get. So at that point, art is what becomes important.

A lot of the time, you’ll see your gut feel play a part. And I say that because you’re the one who’s created this service. You’re the one who knows this inside You are the one who has gone through those different levels of data. You know what this means compared to competitors. You know what the market trends are looking like.

You likely have an idea of what the context looks like. And so your gut feels around what’s going wrong or what might be the reason is going to help you get to that side. So for example, in the scenario that I gave, maybe nothing is wrong with your situation, but your competitor has slashed prices or is doing a one time big Black Friday like promotion. And so all your customers have gone there. And that’s why your sign up rate went down.

It’s got nothing to do with your system. It has nothing to do with any of your matrices. It’s an outside matrix that’s impacting you. But you wouldn’t know that. If you didn’t know that you were meant to look and see and crosscheck if somebody else was doing something else that was affecting your business.

So that’s why I say data is not equal to inside, and then you just always have to try and understand that it’s a combo of art and science, and use that to figure out what the true insight could potentially be. I’ve lost my cursor. I’m sorry it’s constantly happening. Bear with me. Okay.

The next one is, this is perilous. This is my favorite one. I have fallen into this trap so many times where I’m like, oh, I need more data. I need to run some more numbers. I need to do this a different way.

I need to do a top down. I need to do a bottom up. And I just keep going at it and at it and at it, and I’m never really sure that I’ve reached the point where I wanna reach in terms of the data that I have. One way to get about it is triangulation. Now if you go and start triangulation, many different things come up.

This is a term that’s used across a lot of different disciplines, but really what triangulation means over here is if you can confirm a data point through at least 2 different sources, you’re likely closer to the truth than you would be if you were trying to do it by yourself. So for example, if you’re trying to create a business case and you’re trying to estimate what your growth rate target should be in the next couple of years for your business plan. You can do it a few different ways. The first is to look at what the industry is doing. What is the industry growth rate looking like?

If you are a new business, then you will probably have a higher growth rate than the industry because you’re going to grow faster than everybody else because you’re just starting from 0. If you’re a mature business in an industry and you have a growth rate that is much higher than the industry growth rate, then either you’re doing magic, you’re a unicorn, or your number is likely not tempered. Right? You can also look at, say, the growth rate of the biggest competitor. So, for example, if you have an Amazon or a Google in your competitor’s space and you look at their growth rate and you’re higher than that, then something might be off because at the end of the day, they have way more resources, way more ability to invest than you do. So you having a growth rate assumption that’s higher than them is probably not correct.

And then finally, you can also do like a bottoms up where you’re like, okay, this is my baseline and I am going to do a marketing campaign where I’ll spend this much, and I expect I’ll get 500 more customers. I am going to introduce a new feature because of which I will command a higher price. And so because of that, I’m going to get a new type of premium customer.

And so I’ll add 20 more. You kind of build that up and you’re like, okay, this is what my growth rate looks like. So you’ve done it through your bottoms up. You’ve done it, compared to your industry growth rate.

And you’ve compared yourself to a competitor’s growth rate. And now that you have these 3 different data points and you’ve triangulated, you have more belief or more confidence in the data point that you have that this is the correct target to go after. And in that way, you don’t really go into the whole of do I make my numbers more accurate? How do I make my assumptions more accurate? You’ve kind of used 3 different data points to pull it together.

So that you can get out of that paralysis mode and actually go into action. Okay. And then the next one, So this next one, it’s fun. Data is not… I’m gonna bracket the always that this data is not true. Okay?

And I say that because, for example, you’re doing a pricing survey. You go out to a customer, a group of customers, you show them your value prop, and you say, okay, this is what exists in the market right now. This is what I’m offering to you at, I don’t know, $20 a month.

Would you buy this if I gave it to you? And 75% of the customers in the survey say yes, I will. Does that mean when you’re doing your modeling? You should say out of my whole target audience, 75% are going to take the product? No.

Customers can say they’re going to do it because they think they understand what you’re offering, but remember the context piece, they’re understanding your offering from their context. In a value prop description, you might not be able to get through to all of the different points that your product might be bringing to them. And so, really, what is happening is you have a description, customers are reading that description, and then they’re thinking, oh, yeah, this is what it means. And if this is what it means, I would likely buy it.

Are they committing to buy it? No. If they’re offered, will they surely buy it? No. They might not have the disposable income and you offer it to buy. They might not be interested any further. They might have bought a competitor product by the time you brought your product to the market. So when you take that 75% and you straight up apply it, it’s not true anymore. Right? Even pushing the pricing example even further, say you go out to a customer and you’re like, hey, I have this product you can either pay $10, $15 or $20 for it.

What would you like to pay? By default, people are going to choose the cheapest one, $10. Does that mean your $15 or $20 tier won’t work? No. It just means that you have not shown them all the different options and all the different values that the tier can bring.

And by default, the customer has gone into what everybody would do, which is basically choosing the cheapest thing. But it doesn’t mean you have to offer the cheapest thing. You might be able to bring way more value than, say, a competitor who’s offering at $10 and then command that $15. So if you consider that whole data point is your truth, you would be making wrong decisions for your business. So always try and understand that data is not equal to true.

It might be true sometimes. Most of the time it might be. But when all of these, like, context comes into play, a customer’s own filter comes into play, like basic human psychology comes into play, you might end up getting triggers or signals, which will cause you to make the wrong decision. If you’re using just that data as one single point. Cool.

And then, oop, I seem to have forced forward. Can you guys see my takeaway slide? 

S: Yes. 

LA: Okay. Perfect.

Alright. So I said a lot. I gave a lot of examples. And so I don’t expect anybody to take everything away. So I wanted to summarize it all and drill it down to the top three points, which I think is relevant.

So the first one, and I’ve said this a few times, but really if you take one thing away, you don’t even have to take 3, take this. Context is the single most important data influencer. Whenever you’re reading any piece of data, understand that there’s so much context layered into it. For example, in recent times, we’ve had a situation where we had a really important input metric, and we recognized that we were calculating it wrong. And that’s 6 months into the service.

Why did that happen? It was really the context of the person who was there in the beginning who had set this up. And so that person was not a data expert. They did not have analytics support And so they used the data they had at that point in time to set up a formula, which we have been using till now. And we did not realize that the formula was inaccurate.

The context for that person was that they just did not have the expertise to be able to set it up right. The context for us is that we’re so goddamn busy. We never really checked or validated that the formula was right to start with. So the data has been incorrect, and now we’re starting to fix it but at the end of the day, it’s really context. Right?

And imagine somebody who comes into this business 1 year down the line and everybody who’s today managing the business is not there, they will see that matrix go from here to suddenly here to suddenly here. And they’re gonna be like, what? Something happened at that point in time? If you don’t leave that note for them somewhere and who knows they might see it, they might not, then this person 1 year down the line is going to be cool, something really went wrong with the business at that point in time. I should go back and try and figure out what happened, but, really, it was just that context. So understand that context is the single most important influencer. The second takeaway I would say is start with data, but always end with experimentation. Never assume that the data you have is the truth, and so that’s what you should go into the market with. Always try and do A/B test, which is essentially 2 different versions of pricing, 2 different versions of messaging, 2 different versions of features, feature sets, So always try and do that experiment because it gives you way more in terms of what the customer will really do when given these options in real life. It takes away the whole issue that we have with surveys where I was explaining how a customer might just choose the smallest or the cheapest priced option.

When you give them the 2 different prices with the 2 different value sets, they might be like, oh, this value set fits better with me, and so this fits better with my persona. I’m gonna take it. And you’ll see that that higher value tier actually did well. And so that’s why I always start with data, but end with experimentation. Only then take a final call on what should really go up to the masses or what should be scaled.

And then finally, Never forget. You are the secret sauce. Trust your gut, trust your understanding of your business, trust that if something feels off, it is probably off. Trust that anybody who is not involved in the day-to-day business knows less than you, for sure. They might have assumptions.

They might have a better understanding of trends. They might be more aware of that. They might be friends with the competitor and so know what the competitor is going to be doing and so is bringing that value to you or bringing that information to you. But at the end of the day, if you’re the one who’s working on the product on the business line, if you are the one who’s always on the day-to-day, then trust yourself. Trust your gut and always try to read a lot. Reading doesn’t necessarily have to be just books.

Listen to blog posts. Listen to the news. Listen to what the world is doing, where it is headed. Try and get those into you because the more you expose yourself, the better you get at listening to your gut because your gut gives you the right signals. So whenever you look at data, it also comes back to context.

Recognize that you also have your own context. And so try to be unbiased when you’re trying to look at data because a lot of the time, say, we wanna do it a certain way. So for example, I’ll say as a marketer, I’m like, oh, I wanna put this value in. I wanna put this message in. And so anytime a customer says anything positive about that message, I’m like, oh, 100% of the customer said that’s positive because that’s my confirmation bias coming into play.

So you are the secret sauce, but take care of your biases so that you don’t fully fall into the trap of your own context and own filter on the data. And that is actually all I had for today. Thank you so much for listening to me. Hopefully, this was helpful for you, and you take away even one thing that helps you. And if you have any nuggets that I can take away for my own work, I would love to hear them if you can put them on the chat.

Or you could also reach out to VWO with your thoughts, and I will hopefully get to hear them from Sid.

S: Thank you so much, Laura. That was a great talk. We have one question from the audience at this point. What is an A/B test? Can you touch upon that for a bit?

LA: Yeah. For sure. So A/B test basically, it’s literally that. It’s A and B. So you basically do, like, a 2 option test.

And what you try and do is you change only one variable and keep everything else the same. So for example, say, I keep my product exactly the same in terms of the feature. Everything is the same. I will offer it to 2 different types of audience. But I will tell them 2 different things.

So let me think of an example. So let’s use Netflix. With Netflix, I’m offering the exact same product features to 2 different types of groups of audiences. And then with one audience, I’m messaging and saying, hey, this learns from your choices and then we’ll recommend so that you don’t have to think about your next thing. And then with the other one, I’m saying, hey, this is what brings you a huge massive library to your fingertips, without you having to have multiple services.

And you see which messaging works better. At the end of the day, everything else is the same. The product features are still the same. Netflix is actually offering both of them, but as a marketer, when I’m trying to message. I’m trying to see which is the kind of message that really resonates with the customers.

And so that’s the only thing I’ve changed. And you can do that similarly. So for example, you say the intelligent recommendations, piece 1 out. So you take that and this time, you go back again to 2 different groups of audience, and they should be equally sized and have equal persona and everything else. Try to keep as many of the variables the same as you can.

And then you go back with, say, a pricing. So you’re like, hey, would you pay $9 a month, or would you pay $14 a month and see which one wins out? So it’s option A & option B. Change only one variable. Keep everything else the same.

Keep the sizes of the customers the same and see what sort of reactions you get. Does that answer? Let me know if you have any follow-up questions.

S: Laura, I also wanted to ask you. If data is not an insight, then when do you know that what you have is an insight?

LA: This is the fundamental question every marketer asks themselves. They’re like, oh, is this the insight? Is this really true? Back in the day when I was in the telecom industry, I had a leader who helped me out with this answer, actually. His point of view was every time you get an information or a data point, ask why five times.

By the 5th time, you should be at the crux of the or the bottom line of that situation, and you should have that insight. So you ask why five times. So for example, let’s say, we’re saying Uber prices are up, that’s your data point. You ask why? The second level is because there’s more demand, less supply, not enough cars on the road.

And then you ask another why. This is the second one. I was like, why? Why are there less drivers on the road? And they’re like, because gas prices are off, and then you go even further.

And you keep going till you feel like you can’t go any further, and that’s when, you know, you’ve kind of hit your insight. But again, it’s a lot of practice as well. Over time, you will develop a gut to be able to know, okay, I have reached the smallest point I can reach, and there is no other place to go, but practice the 5 ‘whys’.

S: Thank you so much, Laura. We have one more question. How do you control context when you’re trying to recruit for customer research? You can’t possibly know each respondent’s context and how that might impact your decision.

LA: So it’s a balancing act. You want to go as wide as fast full with the different types of customers in your respondents to be able to cover for all different types of context and hopefully it all normalizes. A lot of the time you will have a fixed set of people or type of people that you want to go after. So that’s why I was saying it’s a balancing act. You try to cover as much of a wide audience as possible.

Because at the end of the day, when you launch a product option to the market, a wide group of audience is also going to see and interpret it from their own context. So try and keep as many different types of people into your respondent group, and hopefully that becomes a sample size that’s representative of the actual people out there in the market or the actual customers in the market.

S: We have one follow-up thought here. Thought or question – but it’s probably something… This is the last question that we’ll take. 

LA: Sure. 

S: To give more context, you are suggesting that keep everything the same and just change one variable. It may be the price, target customer group, product size, product design. 

LA: Yes. 

S: Okay. Cool. Awesome. Thank you so much, Laura. This is perfectly well-timed. Exactly 1 hour. And, we are closing it. Thank you. This was quite valuable.

LA: Sorry. I was just trying to thank you for inviting me to do this webinar. It was really nice.

S: We’re more than happy. Thank you so much, Laua. Before we close the session, one final question, Laura, is there any book that you would like to recommend to our audience that you’re currently reading or something that you’ve read, and do you really like and go back to always?

LA: Instead of a book, I’m actually gonna recommend an app. One of my colleagues put me to it. I’ll actually put the name in that chat, it’s called Blinkist. And so what this app does is it takes those leadership books or business books, and its condenses are down to 15-minute to 30-minute blinks it’s called. And so instead of having to go through the whole book, it just takes the main points from the book and highlights it into a very nice little summary.

So I’ve been really enjoying going to Blinkist everyday and reading their new pick of the day. I don’t really choose what books to read. It chooses for me. But it’s been an interesting way for me to read way more than I would have if I had to read the whole book because scrolling through TikTok and Instagram has really ruined my attention span. So that would be my recommendation.

But in general, if we’re talking about this topic, related but not exactly on the topic is the art of oops. Sorry. I’m just trying to type it out into the chat – The Art of Thinking Clearly. This is one book that talks about biases and how you can get away from them.

And so this will be super helpful for you to learn when you are falling into a bias trap and when your context is overflowing into the data. This is one book I definitely recommend. But do check out Blinkist. It’s really good.

S: Awesome. Thank you so much, Laura. That’s some great recommendations there. And thank you to all our attendees who were here. I hope that from this conversation, you have at least one, if not all three, solid takeaways.

And more if possible. Thank you so much, everyone, once again. Thank you, Laura. Have a great day ahead.

LA: Thank you. Thank you, everyone. Have a good day.