Follow us and stay on top of everything CRO

Minimize Conversion Loss in CRO Testing with Multi-Armed Bandits

Duration - 45 minutes

Key Takeaways

  • Multi-Armed Bandits (MABs) can be effectively applied to lead generation campaigns, especially when you have multiple sources of lead generation and want to optimize your budget. The MAB algorithm can handle the budget and automatically test which sources are performing the best, diverting more budget towards those sources.
  • The choice between A/B testing and MABs depends on your objectives. If your goal is to determine the best long-term strategy, A/B testing is more suitable. If your goal is to generate as many leads as possible in a specific timeframe, MABs are the better choice.
  • MABs are scalable in terms of handling more variations compared to A/B testing. However, the insights gained from MABs are not future-proof, meaning they may not be scalable for larger audiences or long-term strategies.
  • VWO is introducing MABs into their platform, offering a new tool for users to optimize their websites and digital marketing strategies.
  • For further clarification or questions about MABs, A/B testing, or other aspects of digital marketing, reach out to speakers who are experts in these areas.

Summary of the session

The webinar, hosted by Shanaz from VWO, featured a deep dive into Multi-Armed Bandit (MAB) testing with Ishan and Anshul, Lead and Senior Data Scientists at VWO. The speakers discussed the exploration factor in MAB testing, which allows for dynamic traffic redirection to better-performing variations. They highlighted the inherent bias this introduces, making statistical analysis challenging.

To mitigate this, they explained their unique approach of applying a simple heuristic to ensure an equal proportion of traffic between all variations, thus reducing bias. The session concluded with a Q&A segment, providing further insights into these testing methods. The audience was encouraged to engage and ask questions throughout the presentation.

Webinar Video

Webinar Deck

Top questions asked by the audience

  • For MAB, what is the recommended size of data to ensure meaningful tests?

    - by Saurabh
    I think MABs can yeah. This is what the benefit of MOB is, but we do not there's no such thing as a minimum number of visitors, but it should meet. Like, in the case of an A/B test to make meaningful ...insight around it. In any way, you can start, like, you would need to deal with it first and the way it's going to work is that it's going, like, if you are implementing Thompson's landing only, in starting, it's going to distribute traffic equally by nature of the setup. And as your test, gains more visitors, whichever variation is performing well. It starts directing that off, and then we'll go ahead and buy design then, we'll start directing that traffic to that winning variation. So I don't think that as such any minimum number of visitors is required in any meeting.
  • How do you see a test being applied to a B2B SaaS lead gen campaign that takes a very long time, sometimes months to reach statistical significance?

    - by Ayala Josephis
    So, okay. I'll make whatever assumptions that I can. And, maybe until you can add to it later. So what I personally feel is that if you have multiple sources of lead generation, and you want to divert ... your budget to the best sources of lead generation. So suppose, if you have $100 for the sake of simplicity, and you have multiple sources of lead generation and you want to optimize those sources of lead generation. You want to optimize what you can do with that budget. That is exactly an MAB use case. And what you can do is you can run an MAB on those different sources. You can set a goal on what leads are being converted and you can let that MAB algorithm handle that budget. It will automatically test out which sources and which routes of lead generation are performing the best and then divert more traffic budget towards those lead-generating sources. So Yeah. That'll be a very valid MAB use case. At a very broad level, I think this problem went to respective there are, like, 2 strategies you have of generating leads. And, eventually, you want to decide which strategy is more suitable in a really long run, and ultimately, your future decisions are going to be dependent upon that strategy. Then A/B test is tested actually your use case. But if your objective is to no matter what strategy is, well, I want to generate as many leads as possible in this particular time, then you should go for MVT. At a really broad level, I think this is how I would decide which algorithm would suit my needs.
  • How scalable are MABs compared to A/B testing or other ML-based methods? What are the average training inference times for a sample recommendation?

    Interesting. So, for scalability, we need more variations than whether MABs are scalable than A/B tests or not. If, really, by scalability, we mean that, then, yes, if by scalability and that should n ...ot be the case. If by scalability, we mean that whatever winners come out of the campaign and whether we can deploy it to a larger audience or not, then I think MABs are not scalable. So like we told that the future the insight that we get out of MABs is not future-proof. So you cannot do that, the insights that you get out of an MAB suppose you run an MAB on a small subset of your visitors and then you want to scale the result that came out to a broader audience. In that sense, MABs are not scalable, I think. So it's not like in case so I think that I kind of understand where he's coming from. So maybe he has this mind of frequentist approach like a tradition with a, ML approach where you Expect that you have a large amount of data, and then only it makes sense to apply an ML. The ML got into it. Now this is because we are using a vision strategy we are incorporating that uncertainty about the system when you have less data, you would always get a distribution that you are not sure of. Whatever result that you will get, it will always be another 13 range. And so in terms of sample wise, there's no ask because you are applying patient you are incorporating that knowledge that you have low samples. So things are going to work out in this patient world.


Disclaimer- Please be aware that the content below is computer-generated, so kindly disregard any potential errors or shortcomings.

Shanaz from VWO: Hey, everyone. Welcome to another session of VWO webinars where experts in digital marketing, experimentation, data, and product, share their trade tips and inspiring stories for you to learn from. I am S ...