Follow us and stay on top of everything CRO

Meta-Analyses in Experimentation: The Whats and Hows

Duration - 40 minutes

Key Takeaways

  • Implement an evidence-based prioritization system for your experiments. This involves using data from completed experiments to rank test ideas based on their win rate and average uplift per winner. This approach keeps the ranking of your test ideas up-to-date based on your latest experimentation results, leading to more successful experiments.
  • Consider the success of different behavioral hypotheses on different pages when prioritizing test ideas. For example, if the social proof hypothesis has been more successful on the homepage than the product finding hypothesis, prioritize test ideas related to social proof on the homepage.
  • Update the win rate and average uplift per winner after every experiment. This ensures that your prioritization score remains current and is based on the most recent results.
  • Add additional attributes to your prioritization model to better align with your business goals. These could include alignment with business goals or OKRs, the percentage of traffic that will see the change, the minimal detectable effect, revenue going through the page, urgency, and ease of implementation.
  • Balance easy and complex experiments for velocity and impact. While it may be tempting to prioritize easy experiments, complex ones can often have a greater impact. Therefore, it's important to maintain a balance between the two.

Summary of the session

In this webinar, Divyansh and Ruben delve into the intricacies of A/B testing and its role in conversion rate optimization (CRO). Ruben, a CRO expert, emphasizes the importance of learning from both successful and unsuccessful experiments. He explains how to analyze data from A/B tests to verify or deny behavioral hypotheses, and how this knowledge can drive revenue and conversion rates.

Ruben also demonstrates how to create a detailed A/B test report, highlighting the difference between results and learnings. He advises against simply summarizing results, instead encouraging attendees to answer key questions to gain valuable insights. These include whether the hypothesis was confirmed, how the results align with previous knowledge, and what can be inferred about customer needs and motivations. Ruben introduces the concept of evidence-based prioritization, which helps build on learnings and successes. The webinar concludes with a poll asking attendees about their current prioritization models. 


Webinar Video

Webinar Deck

Top questions asked by the audience

  • Can you share with us your process to make user research please? And which tool are you using to share with your team, Motion or Myra? Thank you for your generous sharing Ruben.

    - by Hasna
    Got it. That's fine. So tools I use for user research. That can be a lot; I love everything related to polls and surveys, which can be from, several tools like Hotjar. I love Usability Up, which is a ...nice tool, user zoom for remote speed testing, but I also work with clients who have used video apps, even with eye tracking. Indeed for brainstorming sessions, I can use Myra to share with your colleagues, I like that part, be creative, and use your experimentation mind. So see what sticks. If it's sending an email with summaries and works perfectly, if it's a Slack message, we use that. If it's monthly update, lunch and learn, presentations, gamification, or whatever, works, for sharing in your organization. See what works there and experiment in your organization to see how you can best share learnings. I hope it kind of answers to the question.
  • I just wanted to ask if there were some free tools for someone who would like to start, then air something. Sorry. I forgot the name.

    - by Linda
    Airtable. Yes. Airtable is a free version, and that's actually everything we covered in the presentation. You can do so in the free Airtable. So it's very easy, actually. The course is free. Airtab ...le is free. So, yeah. There is a paid option, but you don't need it for this as I displayed in the presentation.


Disclaimer- Please be aware that the content below is computer-generated, so kindly disregard any potential errors or shortcomings.

Divyansh from VWO: Hello. Hi, everyone. Thank you so much for joining the VWO webinar. But we always try and try to upgrade and inspire you with everything around experimentation and conversion rate optimization. I’m your host, Divyansh. I’m a marketing manager at VWO, ...