Innovation and Growth with Experimentation

Haley Carpenter:

Hello, everybody.

D:

We’re starting with the actual discussion. I want to let our attendees know that you too can participate in this discussion. Go to Webinar does not allow me to switch on your cameras, but I can switch on your mics. So do share your thoughts on the questions being discussed. Send me a request using the chat or the questions box from the control panel, and I’d be happy to unmute you.

Haley, please take it away.

HC:

Thank you. I wanna thank VWO for this, but I’m super excited to be here. So I am Haley Carpenter. I am from Chirpy. I’m the founder. And you probably haven’t heard of my brand yet because I just launched my business about 8 weeks ago now. And I’ll talk a little bit more about myself in a bit. So let’s jump into it. And I will also say that I love interacting with people, and it’s a lot more fun and probably keeps more people engaged in awake. If there’s some interaction through the chat and asking questions or whatnot.

So, you know, feel free to interrupt me, and we can unmute you. And I’ll keep an eye on the chat as well for questions. And then if I don’t get to anything during this, I can follow up with you afterward, and then we’ll have a little bit of Q&A at the end as well. But, let’s jump into it. So today, we’re talking about innovation and growth and experimentation, and I am geeked on this topic.

I’m so excited I have a lot of good stuff packed into it. So, yeah, it should be fun. But, this is a little bit more about me. I’ve been in the industry for several years now, and, was at places like CXL, which is now Spiro, and Hand Up in Marketing, which is now Brain Labs. And most recently, I was at Optimizely.

Oh, my big screen is making me look a little bit like Gus. but I quit Optimizely in January and Open Turpey had decided it was time to work for myself, and I’m still staying in that user research experimentation lane and consulting, executing, and training for both of those things, and I love to connect. So reach out. I have all my contact info there, but enough about me.

Let’s get into it. So every business is looking for innovation and growth that’s pretty obvious. Who doesn’t want those things? Right? Like, everyone in the world wants those things, of course.

And so you know, with that, let’s carry that into this, and I’m like to be accurate. So let’s frame the conversation accordingly. So innovation is making changes in something that’s established, especially by introducing new methods, ideas, or products. So this is something that we all do every day all the time. And then growth essentially is an increase.

So we innovate to grow and it’s the cycle and so carrying that forward and thinking about innovation, how does your team decide on what changes you’re going to make? Rhetoric a question or drop your comments in the chat. I would love to hear. Just noodle on that for a second. How do you decide on what you’re gonna do?

Well, unfortunately, a very common answer that I get is something to the effect of well, we make educated guesses, and that happens so much more than you would think. And by educated guesses, that often sounds a lot like, well, we think, we feel, we believe and that’s very subjective language. That’s a very subjective way to make business decisions. So as you might imagine, that doesn’t necessarily lead to the growth that you’re looking for. So we don’t wanna do that.

And then the second piece, when we think about growth, How does your team determine increases? How do you look at performance over time and what does your team try and accomplish? Just think about that again, rhetorical question, or drop your answers in the chat. So for this, a common answer is something to the effect of, well, we’re not really measuring anything, or we have 20 metrics, and we don’t really know which one to choose to base our decisions on. But, you know, at least we’re capturing something.

And not a good answer as you might imagine, or the follow-up or kind of second piece or other answers that I get often is something to the effect of ‘our data is broken’. Yikes. Not what we want either. Sometimes I’ll even have teams like, yeah, you know, it’s been broken for, like, 2 years, but we do what we can. Where they just kinda gave up on it, and now they’re just, again, guessing and going well, we think, we feel, we believe. So all of this is not good. Right? And if you are operating in any of these ways and you are getting gains and you are getting growth. I would argue that maybe you’re just getting lucky, and that is not strategic winning.

That’s not able to be replicated and shared. It’s not sustainable. You don’t really know what is causing what, so we don’t wanna do that. And driving home the point of today, experimentation can be a primary driver of both innovation and making changes and growth to get those increases. So we all want these things.

I said that’s obvious, but experimentation tying into these might not be so obvious, especially if you’ve never heard of experimentation or you’re not doing it or, you know, kind of dipping your toe in. So let’s dive into that a little bit. Experimentation – I listed a lot of problems just now in a couple of different words. But experimentation alleviates a lot of those problems to some extent.

I’m not saying it’s the end all be all. I’m not saying it’s the only thing to do that leads to innovation and growth, but it’s a big part of it and can really be a big part of your strategy and should be. But it helps to eliminate guesswork and validate our decisions, minimizes bias, and mitigates risk to some extent. So I’m gonna talk a little bit more about how that works. And experimentation will lead you to strategic winning and ways that you can replicate and share and sustain and understand. And it will lead you down that yellow brick road to the happy place that you wanna go to.

And with experimentation, I do wanna set an expectation that you’re gonna have tests. Right? And it’s not reasonable to think that every single test is going to win. You will have losses. That’s just a fact. And so it does take some time to ramp up, especially if you’re starting from ground 0, but that is to be expected. And know eventually that you will end up in the green. And so it might be a bit of an up-and-down journey, but if you see this trend line that I have on the right and the graph, you’re still going up with some ups and downs. So it will lead you to where you wanna go, getting you innovation and growth. And at this point, he might say, well, experimentation, that sounds nice, but, like, what actually is that?

What do you mean? Cause I do get that question a lot. And so I pulled a couple of definitions. I like to have a couple to pull from because usually, they’re all slightly different. And then if you mash them together, that’s a better fuller understanding than if you would just look at one alone.

So here I have digital experimentation that is similar, but not identical to the scientific method. So if you think back to science courses that you’ve had and talk about the scientific method, that is being applied here, it’s an attempt to answer a question by establishing a high hypothesis. 

Again, think back to your science classes and then testing that hypothesis and analyzing results, and then delivering winning experiences without that guess worker at risk. So I already kinda mentioned that. So I matched all these together and it speaks to what I’ve kind of been, alluding to so far. And just to quickly give the most basic definition of experimentation and what one individual experiment would look like getting back to your science classes. You have a control, where you’re not changing anything. It’s the existing experience as it is, and we can test on anything really where we own a code base usually, but the most common is, like, marketing websites and products and apps. So think of something from that an existing experience like an existing homepage. Then you wanna know from this example if a green manor background is better than a red one. So instead of just flinging that out into the ether and being like, well, I hope that works.

And then not really knowing, we’re gonna put it into an experiment. and we’re gonna have that as our variation. And then we’re going to split all of the traffic that usually goes to that home page and send 50% to the control and 50% to the variation. Let it run for some amount of time, get a set of results, and analyze that hopefully getting statistical significance, which is another topic, to cover at some point. But, you know, we have those results, and then we can actually see.

Okay. Is the green background better than the red background? Yes or no? And to what extent? So it is really like a scientific experiment where you’re putting rigor behind it and process and structure with systems is a much better way to make business choices. Right? So at this point, you may be like, well, that sounds nice. But how on earth do I go about starting one of these Hailey? Like, is it complicated? Like, what do I even do? What is the first step? Cause I also get that question a lot, which is completely reasonable. And so, before I even tell you about step 1, I wanna start with a precursor that is skipped so often and it will really make or break you and make or break your program. 

So make sure you have enough data volume. If you don’t remember anything else that I say in this, please just, in your mind, if you wanna experiment, like, okay, she said I have to check our data volume. And usually, I call these pretest calculations, and it’s one of the most important things you can do. And it is I would argue one of the hardest concepts to grasp, but don’t let that scare you away.

I promise it’s very doable, and I’ve coached people through it many, many times. But I’ve covered it in the video that I’ve linked to here, so y’all will get the deck. And I’m not gonna go into it in-depth because you can just watch this. It is about 30 minutes You don’t have to give an email. It’s free.

You just go to this link. You watch it, and that’s it. So you watch it. And if your other team members are involved in this and they’re in the conversation with you, have them watch it. And just please have an understanding of how to do this, what it is, why we’re doing it, why it’s important, and then try to do it yourself.

In, you know, for just a couple of pages, a couple of swim links, but go check out this video and I’ll cover it. So now we’ll get to step 1. Assume that you have cleared that data volume hurdle and that you do have enough because that will tell you if you can even test in the first place, And a lot of people assume that any business and anyone can experiment, and that’s just simply not true. You have to have that data volume. So if you don’t, experimentation isn’t for you yet until those numbers make sense, in which case, there are other conversations we can have around research and other ways to validate ideas and a better way to make decisions.

But, for the sake of this example, let’s assume you have enough data volume to run at least one test somewhere, which is the barrier to entry, and be able to just do one. So now, like, okay, Haley, I watched your video. I have at least one place I can run a test. I’m good to go. Now what?

Get a data point. So if you have a place that you want to run a test, and you’re not basing that on any data points, then you’re just kinda back in the same boat a little bit of just flinging stuff out there, hoping that it wins, not really understanding you know, what it’s changing and how it’s changing it. So get a data point and I’m going to explain what I mean by that. She might be like, well, that sounds nice, but what is a data point? And where do I get one?

Research. User research. I say this all the time. I’m gonna be a broken record. But this is a visual to demonstrate what I mean by research and how it ties into experimentation. This is a complicated thing at first glance. I know that. So just go back and check it out later, but the TLDR of it is. The left side is research, the right side is experimentation, and they work together. So where we talk about experiments, there should be an equivalent bucket of conversation around research. Research is where you get your data points to feed your test ideas and support them, and give you more confidence around them. And so you do research. That feeds your experiments. You get experiments done. That feeds your research, and so on and so forth. In this infinity loop on and on and on. And so by research, a lot of people are like, well, what does that mean? And, you know, we have a lot of different methodologies, which is what I typically call them, and people are also like, well, that means? 

Just a bunch of ways that we can do research, user research, and a bunch of tools these days that can help us do that user research to get those data points, those insights from our customers, about our customers or users, whatever you wanna call them, to feed those test ideas, and I’ll give an example of that here in a minute. And so this is Spiro’s model. I don’t need to reinvent the wheel. I’ve used this one in the past. I do love it. So I put it here. But it’s just showing all of those ways that we can get data points, and this might look very overwhelming.

So just start with 1. I’m not gonna cover how to do these in great detail, but there’s tons of content out there to speak more to that and tools and such. So if you like, I don’t do any research right now, pick 1 and be like, oh, a poll. That sounds cool.

Then start kind of looking into just polls. You don’t have to look into all these at once. And then maybe just try that. Maybe try and launch one poll as a starting point and get one data point and go from there. It doesn’t have to be this big super overwhelming thing. But I did say your experimentation will lead you to strategic winning, but also research will lead you to strategic winning, and then you combine those and boom. You’re just you know, you’re on that yellow brick road going to where you want to. So walking through this in a bit more detail with an example, I think a good starting point for research is analytics. That’s what I recommend all the time to clients. and tools for analytics, including things like Google Analytics, GA 4 you may have heard, Amplitude, HEAP Adobe Analytics, and Mixpanel. There are all kinds of analytics tools out there these days, and I would argue that most companies at this point have an analytics tool of some kind.

So fingers crossed that your company already has some of this data, and you might be using it already, which is great. We can tie it into experimentation, and if not, and if you, you know, maybe have a tool and you haven’t seen that data, go ask someone in your company who runs that and manages it, talk to them, work with them, and incorporate it into this, but let’s start here. And an example data point that you would get from analytics would be something like the dropout rate, from the cart to the checkout is 92%. Pretty straightforward. Right? That makes sense and without analytics, maybe you had no idea that was happening. So that’s something we can work with, and I’ll show that in a minute.

But let’s say, you know, we did have analytics, but you want to you’re feeling so and you’re feeling ambitious and you wanna do two methodologies or add another data source like a poll, so then what does that look like? How does that add value when we do more than one? Because typically that’s best. The more data points we have, the better from different methodologies. So how does that help and why is that worth it?

So we have that analytics data point. And then let’s say we ran a poll and we got users who don’t know about the free return policy or the money-back guarantee policy. Those are big things to know, right, because those are good selling points. And so with these data points with one or both, that would be my face. If I found that my dropout rate was that high and that no one had any idea about these policies, so not good. Let’s try and get rid of this face and turn it into a smile. So you have our data points. Right? And now you might be like, great. What do I do with that? We’re gonna create a hypothesis, and I do wanna pause here because I am throwing out a lot. Does anybody have any questions that they wanna throw out at this point? You can drop them in the chat. We can unmute you. We’ll give it just a minute. If not, that’s okay too.

D:

Yeah. I think, Haley, we can move ahead. What’s that? I think we can go ahead.

HC:

Okay. Perfect. So we have our data points. And we’re going to create a hypothesis. And if you forgot what a hypothesis is from your science classes, that’s fine. I’m gonna cover it. So hypothesis format can vary. There are a lot of options, but I have one that I really like that’s if – then – because. So let’s break that down a little bit. Usually, we compare that with ‘we know x’. We know something. So add that to the front. That’s our data point or points. We’re gonna put that in there. Then if we do a, so if we innovate, if we change something, then b will happen. Getting that growth, we’re gonna increase something because of c, which could be because of some psychological principle or because of a best practice. There’s all kinds of things that can go there, but this is what it breaks down into. So we know I inserted our 2 data points. Not gonna reread those. So then, for example, this is how I would turn that into a hypothesis, which is going to inform our test and what we put in that variation.

If you think back to that slide with the red and the green, so let’s say we add content about those 2 policies on the cart page where then we have that dropout, then something will happen. So we’re trying to make those policies more visible, get more eyeballs on them, get more users to see them, and try and decrease that dropout rate. Right? Because the hope is that if they have more motivation, and reduce their fears, we get them to move more through that funnel and hopefully check out and get more revenue. So, that speaks to this where, you know, we’re trying to drop that rate, get more, transactions because of c. 

So here, I just kept it simple. But let’s say friction will decrease due to users feeling there is less risk to place an order. So boom. Now we have our whole hypothesis. It looks long and like a lot, but the font is big. So, now we have all those pieces. And if you extrapolate this out, let’s say you’re doing testing for a couple of months, a couple of years, and let’s say you have, like, 200 tests you’ve run, formatting is super critical because it will make things more scannable, easier for you to find things later, ensure that you have all of the pieces you need. And let’s say, eventually, fifty people are doing this, you have to have some process and some system for people to follow. Otherwise, it’s just gonna be madness and chaos, and you won’t have everything you need, and it’ll be difficult to find things later.

So I have a format. Make sure everyone uses it. And this is just kinda showing if we don’t have all of these pieces, if we don’t have data points, we’re guessing with little to no kind of data involved. You know, we’re still changing something that would stay consistent, but then, you know, if we don’t have metrics and we don’t pick those and we don’t have every data that’s accurate that we can trust, And that other piece kind of goes out the window as well. So, hopefully that reiterates why we want to do all of this. But let’s say we have our data points, and we have a hypothesis.

Now how do we actually turn that, you know, into a test and keep the workflow going? Now we need to come up with a design based on that hypothesis for the variation, for the thing on the right, for us to test against the control. So then after that, let’s say we have come up with a design, usually your design team will do that, then you just send that through the rest of the common test workflow cease in that design to development. You’re gonna QA it before you launch it.

You’re gonna launch it. Just monitor it while it’s running. Wash that data. Make sure nothing goes alright. End it, and then you’re gonna analyze it. That’s about it. Sounds like a lot of steps, but it’s really straightforward, and I’m guessing you already have this workflow to some extent for other things that you’re working on right now. So then let’s say we’ve launched that test. We’ve ended it, and we have our results. An example of that would be, okay, I got a 61%

increase in, let’s say, transactions If that was our primary metric for this test, or let’s say, click throughs from cart checkout, and we increased those by 61%. That’s pretty high. I will set expectations. That’s not usually as big of a lift as you should expect to get all of the time, but for the sake of example, it works. So then if that was my result, I would implement that variation. It will become your new control.

You would stop that test. So a 100% of the traffic would go back to one experience aka that new variation. And then you would try and pivot on that. You would try and take another action off of that. So then, you know, should we make the policies more visible elsewhere also?

Should we announce that more and make it more visible on, like, a category page or a product page and then you turn that into another test and you just keep that going and you’re always trying to improve your performance and get that growth from the innovation that you’re doing. so just kinda coming back to this quickly. making a little more sense out of it. We have our user research through, like, analytics or, psychological and persuasion principles, business context, our archive of tests, if we’ve done it for a while, that we can use to inform new tests, we send that information over into the test workflow that I just walked through and so on and so forth in an infinity loop. So then if you break this out into pieces and think about, you know, extrapolating this out over time and continuing it, for some duration.

Let’s say that adds up to 25 tests in a quarter or a month or a year. You have all of these hypotheses now that you have put through this rigorous process. You get your results, you get your insight, and this is a lot better of an outcome. I would argue then if you hadn’t done this and if you think back to the beginning, you’re just like, well, we make educated guesses and, you know, we think we feel and we don’t really know how it’s affecting things, but, you know, we just hope for the best and look at our broken data. Like, this is so much better, obviously. It’s just clear as day, I think. but lastly, I do want to mention a word about culture relative to experimentation and the business at large.

So if we think about all of the teams that we have in our business, it’s a lot. Right? Like sales, marketing, product, design, engineering, data, customer success, C-suite, so on and so forth. And my favorite way to think about experimentation relative to the business as a whole is in this visual. So I like to think of experimentation and research as a data layer under the entire organization.

So we take those insights in those results and those learnings and we feed them out, up, down, share across, not really maximize the ROI of these efforts. I think Ben LeBay is the one who maybe gave me this outlook on it. I’ve kept it ever since. It’s my favorite. But just to demonstrate that a little bit further, hopefully, the visual is not too small. If it is go back and look at this later, if you want, but let’s say the first scenario is the top image, and we only have one team doing experimentation and say marketing.

And let’s say they run 5 tests in a month. Great. That’s better than nothing. But thinking about culture and how we can incorporate more teams and really kind of integrate this into our company, let’s say now we get 2 teams experimenting and now the product is interested and now the product wants to run tests. Let’s say they do 5.

Well, now we’ve doubled our output and we’ve got double the answers, double the results of the insights, and then keep going with that 5, 10, 15, 20, so on and so forth. That’s pretty incredible. You are really just maximizing the ROIs of the wins and the learnings. And, hopefully, then if you have multiple teams doing it, and this is your business meeting. You’re all sitting around the table.

You come in as the experimentation lead or someone doing it and you’re like, yeah, product. Yeah, marketing. Yeah, sales and you’re all just, like, involved and bought into this and excited and you’re getting the things that you wanna see. Yeah, it’s great.

I’ve seen it happen time and again. And so that’s my note on culture. Just think about it and integrating it more and more and and how you can maybe get more people involved. You know, one person or a couple of people doing it better than none at all, but think about scaling because that helps as you kind of set things up. And then a few final topics I wanted to cover.

Lastly, if you have a lot of tests going, you’re gonna have to prioritize those efforts somehow because you can’t do everything at once. People are gonna have a lot of opinions on what to do, when, and how to do it. And so a prioritization framework helps that and puts some structure around it. And everyone can use this and clearly see what’s going on. I really like the DXL. It’s my favorite one I’ve seen. I’m open to others. So if you have a better one, let me know. Other common ones are pie and ice, pill, things like that. I don’t really like those.

I think they’re a little bit more subjective. this article that I’ve linked to here goes into more about all of these models. So if you don’t have one, go check it out, get a model, have everyone use it, or a framework rather. You know, if you have one and it’s working, great. Fine. But if you have one and it’s, you know, maybe a room or opportunity for improvement, go check out others and maybe adopt it. But, yeah, I like the PXL, so check it out. And so then you have, you know, your prioritized efforts. Let’s say you ran them through a framework. And lastly, with experimentation, road mapping is really helpful.

I’ve seen teams not do it, and it’s a little bit more chaotic. And when we introduce a tool like road mapping, it really helps. And it’s just, a Gantt chart. It’s the best way I’ve seen it done. Not that you have to do it in Excel. This is really kind of bare-bones way to convey a point but with those pretest calculations, you’re gonna know approximate durations of your tests. So you can literally map them out visually across weeks, months, and quarters so that you don’t have unwanted overlap, which is another common, topic of conversation that we could go into a different day. Then you can make sure you don’t have unwanted overlock and unwanted interaction effects. And let’s say you have a swim lane full for 2 months, and Susie Hugh comes to you and says, hey, I wanna do this thing and I wanna do it right now. So then you can go to this, and see if you have an opening.

If not, you can work with Susie and be like, well, here’s what we have lined up. Is it truly that urgent that we need to shove it into our pipeline, or the next opening is 2 months from now on this date if you wanna put it then. And this really helps because also sometimes I get, the eights who are like, oh, I just I have to say no all the time or it’s a real struggle. This just makes it very objective and clear, easy to understand, and it’s not just like you being a jerk. Say no, I’m not gonna do your test.

It’s very methodical, and people usually understand this immediately. Finally, just a few common misconceptions that I wanted to touch on. One is teams will say, well, we’ll just hack experimentation together, and it’ll be fine. Maybe for a while. Maybe. Probably not, but maybe. And eventually, that will catch up with you, though. I promise it does every single time. So whether that’s 3 months in, 6 months in, or a year into it, don’t just hack it together. Please have some structure and intent behind it.

Next, people think like, oh, I just need a testing tool and an idea, and I’m good to go. Right? No. that’s not true either. So you need a tool. That’s for sure. you need ideas. That’s for sure, but you also need to have, like, you know, communication structures in place like where are you gonna talk about this work? Where are you gonna organize it? Where are you going to assign tasks, who’s involved, what are their roles, do you have developer resources, all kinds of things?

So you don’t just buy a tool and then execute. Another one is that you don’t need developers. A lot of salespeople say this all the time. I’ve heard it time and time again, and it makes me very salty and spicy because that’s just not true. So, you know, maybe in Wazzywig editors, what you see is what you get and their visual editors, and you don’t need a technical background.

So some people say to put tests together, there are some tests that you can do without a developer, without technical knowledge in, visual editors, but at a certain point, you’re gonna hit a threshold where you do need a developer. So if that describes you and if that’s kind of something you’re looking for where you’re like, I want something where I can just do it myself and they don’t have desks. Just keep that in mind that you are going to hit a ceiling of what you’re capable of doing, and usually, you cannot do more complicated tests. Especially, if you don’t have server side as an option, which is fine. You know, if you have client-side testing and, you know, you have developers, that’s certainly a level up.

But just in my opinion, and usually a lot of other practitioners’ opinions, you have to have developers at a certain point. Not to talk you out of it, but I just wanna fully be honest about that. And then every business contest, I already mentioned this. That’s just simply not true. You have to have enough data volume. Again, I’ve seen salespeople close deals just for the sake of closing deals, and then you get into it and it’s a major disappointment and people are pissed and upset. So now that that’s just not true and you need to go and watch that video or understand pretest calculations, and then some people are like, this is just too hard. It’s too intimidating. I’m not gonna do it at all.

That’s not true. Please don’t be scared away. I know there’s, like, maybe a lot of big terms and acronyms and stuff, and stuff seems really technical to some extent, but I promise you it’s very doable. It’s very easy to learn about and figure out, and there are all kinds of resources these days and people that can help. So it’s not too hard.

On the flip side, some people like, well, it’s, you know, it’s just so easy. Anyone can do it. Again, we’ll just hack it together. That’s not true either. Just be somewhere in the middle with your expectation that you know, it’s gonna take some work and sometimes it might be hard and sometimes it might be really easy. But we’re not on either one of those extremes. So also, just the TLDR, please try it. Please do this. Please look into it. If you’re not, you know, or if you do have experimentation already, try and level up maybe in some way.

I don’t know. I hope you can see a video on my screen. If someone would maybe give me a thumbs up, or not, I’ll just roll with it. But can you see my video?

D:

Yeah. Yeah. We can.

HC:

Okay. Very good. So this is scrolling through VWO’s success story page. And, VWO is an excellent tool. Like, I was talking about to experiment, you have to have a tool of some kind.

And so that is what VWO helps you with among, other things, but they look at the success story. Like, just pages and pages and one after the other of amazing lifts, in all kinds of awesome metrics across different companies in different industries and different company sizes. So, you know, all these companies have the data of volume. They cleared that hurdle, and then they have their programs going. They’ve actually done the work of getting the tests out, and it was totally worth it.

In VWO, like I said, it’s awesome. So investigate this as an option for sure. Make sure you talk to VWO. And, you know, hopefully, you end up on a success story page one day and you’re turning to a case study with these awesome results. I did just quickly zoom in on too specifically.

And so, I wanted to call out, like, some big ones and show also, like, real variations that teams have run. So, you know, it can be simple where you do literally just change one thing, like in the examples I gave where it’s, correct or versus a green background. I can go into this in a lot more detail, but, I don’t have time here, but look at this. Like, we’re adding a whole section. We’re adding icons. We’re adding copy. We’re speaking to benefits and what makes us different and why you should buy. It’s a lot of awesome change at once. And look at the list that it got and look at the data now that we’re able to put behind this change to validate it and show that we should do it and show that it’s worth it. This kind of stuff is awesome.

And this one, they changed an entire page. Look, it’s totally different. This is fascinating stuff to do and to test. And usually, drastic changes like this can get you an awesome lift. So look, they got a 12% jump and lead conversion over the control.

That’s awesome. You know, so be a case study. Do experimentation. and then just to recap, I’ve already covered all this, but it’s here for later. So, you know, is experimentation worth the hype?

I think it’s pretty obvious to me. I might be a little bit biased. But, yeah, hopefully, I can see you to at least look into it, but that’s it. So I think we can do some Q &A now. Oh, we can’t hear you.

D:

Thank you for the awesome presentation, Haley. Some of the examples I personally do, and, I can definitely vouch for the tips are going to help experiment us across the spectrum. We have a few questions, and I’ll be unmuting you, Velasco. Please go ahead with your question. And, Yannie, then you can probably, go ahead next.

Palesca, I’ve unmuted you. Please go ahead with your question.

Palesca:

Oh, so basically what I want to know is if I change something, for example, in an app and an onboarding page, I do some kind of changes, and this can also impact my conversion and in the payment page or something like that. Right? That’s the idea that I have. So if I want to test things like in the onboarding and payment page, how do I do that and make sure that I only look into onboarding and not only look into payment?

HC:

Good question. You open a can of worms, so I can talk on this for a while. I’ll try and be fast. So this is actually a bit of a divisive topic among crack conditioners, and it ties into what should my primary metric be for my tests to get the best understanding of how it performed and how I should take action on it after the fact. And I will say that you can run tests across an experience like let’s think of a website.

You can do a homepage test with a PDP test, a category page test, and a cart test. You can run across different swim lanes at once. Just typically do not run more than one test in the same page at the same time, and you have to have a little bit of a strategy to when you launch those tests. if you are running across an experience, or you can have one test where you’re changing things across multi-page, multiple pages at once there are different ways to go about it. But kinda I think more on the measurement piece, a little bit, is what you’re getting at.

What would be the primary metric is a question that comes to mind. And so some people say, well, it should be the action closest to the changes that you’re making. And some people are okay with engagement metrics like button clicks, click-throughs, pages, video views, and things of that nature. And then that’s in contrast to, should it be down funnel metrics that are more, about the bottom line, like form submissions or transactions or revenue, I’m very opinionated that most tests, regardless of where they’re happening, you should still use down funnel metrics because that’s what we care about. That’s what most people in the business care about, especially when you go to justify your program and your tests and the resources that it took. To be able to do all of that. If let’s say you’re presenting to a CMO at the end of a quarter and you’re talking about tests and they just wanna know the bottom line if all of your tests are based on those higher funnel engagement types of metrics, they’re probably not gonna buy into that as much. And I’m like, great. I love, you know, the button clicks, but, how is that impacting my bottom line at the end of the day, and how does this work?

So, does that help a little bit?

P:

Yeah. Yeah. It does. but it also comes to another question, if I change a lot of things at the same time, should I test everything at once or should I consider it with the product team? Like, does it work to do everything at once, or should we do one test at a time? First, test the icons, for example, and after testing the texts and things like that.

HC:

I love this question so much. So it depends as that’s the answer for a lot of things. One part of it is the pretest calculations. So you’ll hear me talk about something that’s called minimum detectable effect for MDE. And some of that pretest calculation tells you how dramatic of a change you need to make in your tests to give you the best chance at statistical significance, and that’s kind of in parallel with estimated durations.

But essentially, there’s a sliding scale. So some tests, you have the wiggle room based on the data volume to do smaller things where the idea can be a little bit smaller. Like, maybe you just change one line of copy. That’s what I call a small test, in one place. If you have the freedom to do that, great. Or, you know, some tests, you do wanna swing bigger and you wanna change 5 things at 1, and do big things all at once. Any direction on this scale is fine. You just need to find out in those calculations what you’re working with.

and where you need to fall on that sliding scale, on my camera’s being weird. One thing to keep in mind is that typically the more changes you make at once there’s a trade-off there, of where you’re aiming more for impact because the more changes you make at once typically you have a better potential for a bigger impact. But you’re losing out on the precision of learnings, where if you go on the other side of that and you make one change, you’re increasing the precision of learnings, but typically you’re dropping the potential for impacts, which is also something to think about. Does that help?

P:

Yeah. That helps a lot. Thank you. Really appreciate that.

HC:

Yeah. Yeah. Awesome. Good. I think there was maybe one more question.

D:

Yeah. Yeah. Definitely. Yani, I’ll be unmuting you. Please go ahead with your question.

I’m sorry. that’s how it’s pronounced, right, Yannie. So, yeah, you can go ahead.

Yannie:

Okay. Thank you. Yeah. Haley, thank you so much for this presentation. My question is, could you talk more about that pretest planning that you mentioned earlier? Thank you.

HC:

Yeah. So that is a big topic that I definitely won’t be able to cover even in the next 5 minutes. So I really would encourage you to go check out that video. But as far as data volume, I know I keep saying that term. Really what I’m talking about is, traffic, whether you define that by sessions, users, MAUs, whatever that means to you in a given use case, that paired with conversions for your primary metric whether that be something more down funnel, or something higher funnel. I will say, kind of circling back quickly to that perspective I mentioned down funnel versus higher funnel is usually more of a conversation I have with marketing teams. If we’re talking about a product, it kind of changes entirely. So there is that caveat, but, data volume, traffic, and conversions. There are different calculators for this and different ways that it can be kind of approached. For example, you probably have seen some sample size calculators where that’s the output. There are some calculators where the estimated duration is in MDE or the output. I have very strong feelings that sample size calculators are useless and not really the way to do it. You can back into stuff that way, but it’s just annoying and why do that? Really, you’re just looking for the duration estimates and the MDE.

Those are the 2 common big data points that you need to look at those. But, yeah, go check out that video. But does that add a little bit of context? Does that help? Do you have any other specific questions that I can maybe just quickly touch on?

Y:

No. That is all. Thank you so much.

D:

Yeah. Of course. Any other questions? I think that’d be it. Do reach out to Haley. I’m sure, just a second. Let me. Yeah. I’m sure she’s going to be of great help in setting up your experimentation and actually accelerating the results and kind of solving all your doubts. She has been kind enough to do this presentation and it was an awesome presentation, Haley. Thank you so much. And as you know, the question answers are kind of proof, let’s hopefully collaborate again sooner. Yeah.

And to everyone, if you’re still on, I would love to chat more or, connect. So feel free to reach out to me. And if you wanna talk more, if there’s something I didn’t cover, ping me, DM me, and I’ll be happy to talk.

Yep. It seems like there are no more questions. Yeah, do reach out to Haley. We’ll be happy to redirect you. Thank you so much, Haley, for this presentation. Thank you all for attending. See you soon for another VWO. Bye.