- Maintain momentum in your CRO program. Consistency and regularity are key to ensuring the program's success and delivering a positive return on investment.
- Communicate the value of your CRO program effectively. This includes regular updates to stakeholders, highlighting the benefits and improvements made through the program.
- Use tools like Slack to provide concise, fact-based summaries of your CRO program's progress. This can help stakeholders easily understand the value being delivered.
- Frame the resources required for the CRO program in terms of the value they can deliver. For example, if more development time is needed, explain how this can lead to greater success and growth for the program.
- Adopt a mindset of testing to learn, not just to win. This approach can lead to better win rates and more valuable insights from your CRO program.
Summary of the session
The webinar, hosted by Vipul from VWO, featured Chris Marsh from Dash of CX, an expert in UX, data, and CRO. Chris shared his insights on experimentation and CRO, including how to decide whether to run a test based on traffic or conversions. He suggested soft launching tests to 25% of traffic if stakeholders are worried, but ultimately aiming for 100% of relevant traffic.
Chris also addressed questions about testing on low-traffic pages, recommending using one variation and earlier funnel KPIs. The session was interactive, with audience members actively participating. The webinar concluded with an invitation to connect with Chris on LinkedIn for further discussions.
Top questions asked by the audience
Sometimes we see some experiments that are not good enough, maybe for 10% of the users that come into the website. So in some talks with the team, sometimes we try to identify if it's okay to run the experiment for 100% of the users. Maybe in the production side or maybe in the tool. But, yeah, basically, my question is: Do you think it's possible or okay to run the experiment for 100 percent of the users?- by DiegoHi. Yeah. Thanks for the question. Yeah, if all of the users are relevant, then you should run it to those users. Or I mean, if your program if there's, like, high risk to the test and you know, maybe ... it's on the checkout or something, then you could soft launch it and you could launch it to maybe 10% of users or 20% of users to start. And then if the software allows, you either ramp that up to a 100% or you quit that test and relaunch it to a 100%. Because having more data just gives us more accuracy on the results. In fact, for one client, they're UK only, and they only sell to UK shoppers. So I actually target only the UK for those experiments. Because the other data will just produce noise. You know, it might balance out, but there is unnecessary noise. So it's worth thinking, you know, usually it's to a 100% of relevant traffic, and usually it's to all audiences when I test. But, yeah, those, hopefully, those are some good thoughts for you to consider.
A lot of times, we'll wanna test things that just don't have a ton of traffic to the pages. So we're stuck with really long test times. Do you have a method that you normally go to? Like use ad words or something like that to try and drive traffic, when there's not a ton of it? But these are high value pages. We just don't have enough visitors.- by DavidSo there are a few things you can do. You could try driving more traffic, but that’ll get expensive and will you be able to continue doing that afterwards? The thing is the way I look at it is I wou ...ld recommend only one variation, unfortunately, because that will help. So you're getting more data into the test. If you use more variations, you need more data into the test. So, one variation helps. Look at your KPIs, the measurements, and you can use an initial kind of action on that page rather than measuring further down the funnel. You could measure the progression on that page. So perhaps just click on the link that's the most high value link on that page, like add-to-cart or such or a few products. So you could use the KPIs earlier in the funnel. And also, you know, you do want sort of 300 plus conversions per variation. A daily 500 or 800 or 1000, per variation at the end of the test. If you don't have that, you’ll usually simply have to hold your hands up and say, we can't test this. But that's not so bad because if we're doing the CRO research first, then we have signals to say that this is useful. And as I've explained, there's still a 20 to 30% chance that these changes will actually improve your KPIs if we're using research to get to these solutions. So also you could use research to validate the solution. You could use a test on it. You could launch it and then run heat maps on it, look at sessions and things. There's these kinds of ways. If you wanna message me on LinkedIn, I can follow-up. I've created a list before about my ideas - there are 5 or 6 ideas. But that's most of them. Hope that helps?
One of the points that Chris mentioned, which was a very good point that we should create our hypothesis from data. So, I always have challenges with this. So my question was how do you use data to create the hypothesis, the percentage and the absolute variations? What data point did you look to come up with the hypothesis to test the percentage in the absolute hypothesis?- by KudaFor the example I used. Yeah. So we used survey data. So we polled the audience on the site and asked what are the most important factors to you when shopping for perfume realistic, like, 7 or 8 facto ...rs. And 3 were by far the most important factors. So have you heard of the PXL prioritization method? I recommend using that and you just kind of like, so how do you relate it? So well, this is why our hypothesis is quite tricky to use. I mean, it has the format of so for this example, I would say if we change the product page, to show the pound value. So it starts with “If”, “then” the shopping KPIs will increase or conversion rates will increase. And then the final part of a simple hypothesis is “Due to the reason”, and that's where you add your idea, your theory as to why it will happen. And I would say due to better showcasing low value or great value on the product which is important, shown by data.
Can we also add the target audience as well in that structure of the hypothesis?- by KudaYeah. Yeah. Exactly. You can flash it out at the beginning, perhaps you add. So it's defining the audience at the beginning. And this is a thing with the hypothesis that they're big and convoluted, bu ...t they are useful. So if you can use them, that's great, but people skip them often. And they're, you know, they don't give a snapshot. If you just look at an A/B test idea and you have to understand the hypothesis, you have to sort of study it for a minute and think about it. It's like how-to-digest it. It's not that intuitive, which is where the PQ framework makes things intuitive and sort of flips things around and lets us have a problem-focused approach and a question-focused approach as well. But, yeah, if you can use hypotheses, if it's working well and you're using them correctly to create tests, which help you learn, then you can stick with that. But what I've found in the past is that hypotheses don't always help you do that. It can be weirdly easy-to-go, of course, and to create a solution. And then it's hard because it's hard to look at the mock-up solution and look at your hypothesis, which is quite hard to sort of conceptualize and check that they fit. Basically, if you're making too many multiple changes on the variation, you're not gonna get accurate learning. That's where it can go awry. And it might not be that obvious that it doesn't fit with the hypothesis, but it won't.
I wanted to find out when you're trying to run a test, and you want to find whether you have enough traffic, should you basically base it on the number of conversions on the page or the traffic on the site? What should be the best that you should use to decide whether this should be a test or not? Is it traffic or the number of conversions?- by PhilemonHi, Philemon. Good question. Ultimately, it's the conversions, but they do relate. So if your conversion rate is higher, then I think it means that your test can run a bit shorter. So they do inter ...twine somewhat, but ultimately, it's really important to get enough conversions. So we want over 500 conversions, around 4 weeks per variation. And even then, I would use this kind of insight with a pinch of salt, whereas if we get over 1000 conversions, then the data is even more accurate. But you can use calculators, and it's best to use the calculator before you run the test so you know exactly what you're looking at. And the calculator will tell you what uplift you need, which is really useful, because you want about 5% to 10% uplift. If the calculator says you need 20% uplift to run this test, then it's not a good test. You could try and use a different KPI, which requires less uplift. Because it's hard to get 20% uplift or more often.
So Good They Can't Ignore Youby Cal Newport
the author talks about making small bets in your career. Okay. So he says, make small bets, and it maximizes your chance to win. Small bets are great because they're bite-sized, you know, and they're just simple, simple, experiments, simple tests, simple things here. He says, the main thing is you're learning as you go. He says, one day, one of these small bets, it will pay off huge. What's up, man? This reminds me of that CRO program. We should have been making small bets. I thought, why didn't I hear this 2 years ago?But here you go. So to summarize, it's so important to build and keep momentum. Now the second CRO program, which faltered and hit an end, was a US store selling discounts, daily deals, lots of items for the day on a deal.
Sprinting to Show Valueby Rommil Santiago
If you want to grow your CRO program, there's a great chance to do it. So you can state the hours that were required to achieve these results. So we can say - hey, this uses 20 hours of dev time per month. We would love to unlock more dev time because that might be the blocker and we know that we can grow the program if we unblock that dev resource. So this is where we can do it. We can tie these resources back to the value and success and say, you know, if we add x amount of hours of dev resource, we can achieve this growth to our program and to our success.
Disclaimer- Please be aware that the content below is computer-generated, so kindly disregard any potential errors or shortcomings.