Follow us and stay on top of everything CRO

Testing Big vs. Testing Small – How To Evolve a Product Rapidly Without Sacrificing Learnings.

Duration - 35 minutes

Key Takeaways

  • Consider the trade-off between granular learning and evolving fast. While Multivariate Testing (MBT) provides granular learning, it requires more traffic and a higher threshold of significance, which can slow down the evolution process. Bundled experiments, on the other hand, can potentially help you evolve faster.
  • Be deliberate when bundling your hypotheses into experiments. Identify which hypotheses need to be isolated due to high risk or curiosity and which ones need to be bundled due to development dependencies or lower development effort.
  • Use a controlled approach when bundling. Identify the "need to's" - hypotheses that need to be isolated due to high risk or those that need to be bundled due to development dependencies or lower development effort.
  • Be honest about your bundling needs. Sometimes, what seems like a need to bundle is actually a nice-to-have or an order in which things need to happen. Understanding these upfront can help you make better decisions.
  • Use your weighting system to guide your bundling strategy. The stage you're at in your process will dictate how aggressively you should bundle your hypotheses.

Summary of the session

In this webinar, Iqbal Ali, Optimization Specialist Consultant at Mild Frenzy Limited discussed the balance between “Testing Big vs. Testing Small”. He addressed the common question of why not to use Multivariate Testing (MVT) for everything, explaining the risk of false results due to multiple variations and the need for high traffic and significance. He suggested bundling experiments based on their goals and themes, and setting limits on the number of hypotheses.

Iqbal also highlighted the importance of identifying ‘need to’ experiments that require isolation due to high risk or curiosity. He emphasized the need for a controlled approach to bundling, considering factors like development dependencies and lower development effort. The webinar aimed to equip attendees with practical strategies for their own CRO programs.

Webinar Video

Webinar Deck

Top questions asked by the audience

  • How do you or your team decide which tests should be bundled and which tests should not be bundled? Is there any specific framework that you follow, or are there any parameters?

    So with tests, I think it's important to go through the process and split them into hypotheses. So go back to the basics of what you're trying to learn from each test. And split out all of the assumpt ...ions that go into it, the hypothesis, and then just go through this process. They'll just systematically go through, what are the variables, what are the learnings you wanna get, into how much risk you want to take and then we can, bundle them from that side.


Disclaimer- Please be aware that the content below is computer-generated, so kindly disregard any potential errors or shortcomings.

Shanaz from VWO: Hello, everyone, and welcome to another, webinar with VWO. Today, we have with us Iqbal, CRO lead at Trainline, currently working as a consultant and trainer. We’d like to apologize for starting thi ...