Follow us and stay on top of everything CRO

User Research: The Superpower Behind Experimentation Programs

Duration - 50 minutes

Key Takeways

  • Embrace change and experimentation: Making significant changes to your website or product can lead to substantial improvements in conversion rates. Don't be afraid to test new ideas and approaches.
  • Use data to validate changes: Whenever you make changes, use data to validate whether those changes are effective. This can help you understand whether your changes are worth the effort and resources.
  • Consider the impact of changes across the entire user journey: Changes made on one page can impact conversions on another. When testing, consider the entire user journey, not just isolated pages or features.
  • Prioritize down-funnel metrics: While engagement metrics like button clicks and page views are important, prioritize down-funnel metrics like form submissions, transactions, and revenue. These metrics are more closely tied to your bottom line and can provide a clearer picture of the impact of your changes.
  • Don't run more than one test on the same page at the same time: This can lead to confusion and skewed results. Instead, have a clear strategy for when and where you launch your tests.

Summary of the session

The webinar, hosted by a representative from VWO, featured Chris, an expert in user research and experimentation. The discussion focused on the importance of both qualitative and quantitative research in understanding customer behavior and uncovering opportunities.

Chris emphasized the value of usability testing not just for assessing prototypes, but also for discovery research. He also highlighted the need for analytics to understand the scale of a problem and determine its priority in the business roadmap. The session was interactive, with attendees asking insightful questions about user research methods and experimentation.

The webinar concluded with a Q&A segment, with the host promising to forward any unanswered questions to Chris for follow-up. Attendees were also encouraged to connect with Chris on LinkedIn for further discussion.

Webinar Video

Webinar Deck

Top questions asked by the audience

  • Which product or platform is best for moderated testing, other than the coffee shop approach?

    - by Jimmy
    Well, we actually use look back. And that's the software we use. And it works pretty well. There there there's always some so we used to do everything in our labs in London. And then COVID came, and t ...hen that forced us to do everything remotely. And, actually, at the time, we were thinking, oh my god. It's kind of like ruined all our use beta testing. but in fact, tools like loopback made it possible to do remote testing and still keep it moderated There are some challenges with this because people have to download something onto their device and get it working. But overall, the pros outweigh becomes a of it. And on the, the good thing is you still get to see the picture in the picture. You can see their face, like, on the bottom left here, and you see their screen as well. But there are just a few issues on iOS. Sometimes you can't see their face, etcetera. I think the best thing with it, Jimmy, is that you can anybody can then log in and watch and write notes, which are on the right-hand side of that small screen. So you can write notes and, comments and things without the user knowing. So, actually, you can get your whole team involved watching everything. And then the notes are time-stamped. So when you want to create a video, you can just easily create that clip of where you took that note of that observation. So that's a really good tool.
  • Do you have any top tips for good screener questions just to make sure you get the right type of users within the moderated tests?

    - by Jimmy
    Yeah. That's a good question. So the the one that's I mean, actually, we we want people who haven't done much research before. So in actual fact, one of the well, you have to I'll ask it in the right ...way, but we don't want people who have done research more frequently than in the last 6 months or 3 months. so that's an important one. You have to watch that sometimes the problems with the panels out there. there are tools that make it really, really easy to run more unmoderated tests, but the panels have users who are just doing tests all the time. And you can kinda start to see it, actually. You can start to think but they're almost ready with their opinions. They almost said, oh, yeah. That color shouldn't be blue anymore. And, of course, we don't really care what they think in terms of design, we just want them to do that journey as if they're a real shopper and just go ahead and and find a hotel or buy that dress, for example. the other one which is really good is don't recruit people who have used that product or website before. So if you're a retailer, for example, make sure there's an open question where they will say where they've shopped and where they haven't. And then you can make sure that they're not a regular customer because a regular customer of your website will have already learned all the problems with it. So they're they're They're not the best person to use. But, also, you can put a list of light competitors, and you can recruit the ones which recruit which use your competitors. and then they're they're perfect target audience. That'll make sense.
  • When you experiment for a long time we have seen a lot of things that work and don't work. In an experimentation culture you form this notion that, okay, might be this is going to work, this is not going to work, maybe I should add a pop-up here, or I should remove this. So how to prevent your team from reaching that saturation? But, okay, yeah, let's do this, and let's get this client over with, you know, let's not do anything further. So how to eliminate this, generalization or saturation from our teams per se?

    No. That's really good. That's almost like, if I'm hearing you correctly, it's almost like they're solutionizing first, aren't they? You can kind of get this. Some people will all jump to the solution ..., and it's just a generic solution. It's like making everything sticky, which seems to be a trend at the moment. Make the button sticky, make the sticky, make the sticky, you know, but a very jumping. Again, it's jumping to a solution. I think that the trick is as soon as you get people to think about what the problem is they're trying to solve, it gets them away from that way of thinking it gets from from just jumping to jumping too quickly to an AB test. And that's what that's the power of research. I'd say, so there are things like making sure everybody actually has in a test plan, a customer problem, and also making sure it is based on real evidence, not just something made up or this kind of will just quickly go and find some data to back up my idea type of thing, which I mentioned earlier. Those types of methods can help people get away from that kind of solutionizing way, and culture.
  • Which would be your preferred method of user research if you wanna leverage the volume of feedback over the quality of feedback? There'll be surveys, for instance, you can get volume, but in interviews, you can get pretty greater insight.

    - by Alex
    Yeah. That's a really good question. I think the, it's always the best it's always best to do both. That's why I go on a lot about the quantum, the quant. but it's good to get really close to the cust ...omer, to get rich insights, to understand why things are going on as well as to observe their natural behavior. So that's why usability testing is so good. unfortunately, a lot of teams only use it for assessing a prototype not almost assessing the idea or validating the idea they already have. What they don't use it enough for is more discovery research for uncovering opportunities in the first place. So that's really good, but you also need to have the quant site. So you need to then, use tools like analytics or behavior analytics to help you find out what scale of that problem. So so for example, or just the number of users who are going through the problematic journey, and then that will give you a real idea as to whether you should focus on that in your road or not. So it's it's that combination, but definitely really face-to-face research with with target audience. And you don't need to do many, like I said, you can just do 2 or 3 days is plenty.


Disclaimer- Please be aware that the content below is computer-generated, so kindly disregard any potential errors or shortcomings.

Vipul from VWO: Thank you so much for tuning in live for this yet another insightful session of VWO webinars. As always, my name is Vipul, and I’m the senior marketing manager at VWO, a full-funnel experimentation p ...