Follow us and stay on top of everything CRO

Holiday A/B Testing Strategy: How Invesp Generated $37 Million During Holidays

Duration - 60 minutes

Key Takeaways

  • Understand the problem: Conduct discovery research to understand customer confusion, unmet needs, expectations, priorities, objections, distrust, anxieties, and unique user behaviors. This will help you identify areas for improvement and innovation.
  • Quantify the opportunities: Use quantitative research and analytics to understand the extent of the problems and which audiences are affected. This will help you prioritize your efforts and resources.
  • Use effective research techniques: Utilize usability testing, user interviews, post-purchase surveys, call center insights, and sales team interviews to gather valuable insights. These techniques can provide rich and powerful ideas for improving your product or service.
  • Balance qualitative and quantitative data: Use a mix of both to build strong problem statements, inspire creative ideation sessions, and develop data-driven hypotheses for tests. This will lead to more effective solutions and higher overall success rates.
  • Apply insights to your job: Use the insights and lessons learned from the webinar to improve your own work. This can help you better understand your customers, improve your product or service, and increase sales.

Summary of the session

The webinar was hosted by Sid, a regional marketing manager for VWO, who has been with the company for four years. Sid has a rich background in A/B testing and experiments, which he has been conducting for nine years across various organizations. He introduced the guest speaker, Khalid Saleh, a veteran in conversion and optimization since 2006. Khaled has run over 25,000 A/B tests with Invesp, generating upwards of $37 million in revenue. 

Khalid, who was a software architect in his previous role, began his presentation by sharing a story from 2017. He received a call from Karim Ozguai, the then VP of Marketing at Z Gallerie, a large furniture retailer in the US. Karim had asked Khalid if he could help improve their conversion rates during the Black Friday and Cyber Monday sales period. Despite the short notice, Khalid accepted the challenge, excited about the opportunity to work with Z Gallerie.

Webinar Video

Webinar Deck

Top questions asked by the audience

  • Have you seen any winners that only performed well during the holiday season, except, of course, holiday-specific visual slash coffee?

    Sometimes, it's a good practice that the winners that you have during the holidays are away from yeah, I have a special offer. I'm not going to have it anymore. Correct? So that goes away, but it's go ...od practice to run some of these tests after the holidays. The reason is the holidays is a highly motivated visitor solicitor with a correct card in hand. Correct? So that's motivation, and your barrier for somebody who's doing conversion optimization, trying to improve conversion rates becomes harder post-holidays. So sometimes a good practice away from the holiday-specific offers is to go back and run tests. But remember, also, the holiday testing is very focused on what is on messaging and offers and urgency and scarcity. That's what you're testing. We're not I showed a couple of examples where we're like, hey, and then this is actually a good example where for this, the car that sold the car, the truck beds where it performed really well, performed amazing during the holidays, a 30% lift, and it still increased conversions. If I recall correctly, it was about 11 or 12% post-conversion. So it's still really good, not that the 30% that we were really, like, you know, celebrating during the holidays. most of the time, you will see was difficult to say I would run the tests again on the experiments post-holidays, to see if I am seeing consistent results or not.

Reading Recommendations

  • Competing Against Luck

    by Clayton M. Christensen

    This book was recommended for its insights into customer psychology, decision-making processes, and strategies for smaller companies to compete against industry giants. It was specifically recommended for its easy readability and practical advice.

  • CXL Blog

    by CXL

    Despite being a competitor, the speaker praised the quality of their content.

  • VWO Blog

    by VWO

    The speaker mentioned this as another valuable resource for deep content.

  • Invesp Blog

    by Invesp

    The speaker mentioned their own blog as a source of deep content.

  • LinkedIn

    by LinkedIn

    The speaker also mentioned LinkedIn as a platform where a lot of free knowledge is shared. He also mentioned that they have written a book about Conversion Rate Optimization (CRO), which is an Amazon bestseller, but did not provide the title of the book. For more detailed information and examples, the speaker suggested emailing to get a copy of their playbook. He also encouraged the audience to connect with him on LinkedIn for further discussions.


Disclaimer- Please be aware that the content below is computer-generated, so kindly disregard any potential errors or shortcomings.

Sid from VWO: Hello, everyone. Thank you for joining in, right in time, and a little ahead of time as well. I’m Sid here. I’m gonna start, right away so that folks can join in as we go along. I’m startin ...