Your checkout team ships a winner. Your search team launches a hit.
The individual test reports are glowing green, but when the quarterly review hits, you see little to no impact on your revenue.
Why? Because your “winning” updates might be silently colliding, and there’s no way to know if they are.
You are left wondering: “Is our product roadmap ‘actually’ delivering the impact?”
Enter Holdouts.
Now you can exclude a small, intentional subset of your traffic from a series of new feature rollouts, experiments, and personalization campaigns, over weeks, months, or quarters.
This subset acts as a pure, stable baseline and represents exactly what user behavior would look like if you didn’t deploy any product changes.
How to use Holdouts?
Setting up a Holdout in VWO is seamless and fits right into your existing workflow. By associating a holdout with your feature flags, it universally works across all your feature rollouts, personalizations, and experiments.
We offer two types of Holdouts:
1. Global Holdouts: Automatically exclude your Holdout users from all new feature flags across your product to measure your entire roadmap’s impact.
2. Selective Holdouts: Manually associate specific feature flags to your Holdouts to measure the combined impact of a specific theme or initiative.
Once active, you get a definitive, dashboard-level view comparing total visitors, conversions, and primary metric values between the Holdouts and users bucketed in your campaigns overall.
How does it help your industry?
- For E-commerce & Retail:
Imagine you launch guest checkout, revamp your recommendation algorithm, introduce an enhanced search experience, revamp the product listing page, and evolve the search experience, all in a single quarter. Individual A/B tests for each, can’t prove the cumulative bottom-line impact. By keeping a set of users that are isolated from these releases, you get scientific proof about these releases, whether they are even increasing your conversions, AOV, or revenue. - For SaaS & Software:
Move the conversation from “Did a specific onboarding change win?” to “Did our Q3 product roadmap actually increase user retention and Customer Lifetime Value?” You finally have the data to prove your strategy works. - For News & Media:
If you launch a new personalization algorithm for the news feed, a metered paywall, and start testing a new ad platform, make a few UI/UX changes, and revamp article writing style, in a single quarter. Such individual A/B tests can’t prove cumulative impact on engagement & revenue. By having the Holdout users as your original baseline, you get a clear indication of the impact of your launches.
Why is this important to you?
Justify your product investments and illustrate ROI:
Solve your leadership’s biggest struggle: “Tangibly understanding the ROI of product development.” Deliver a data-backed analysis to prove exactly how your development efforts are impacting key business goals.
Proactively detect anomalies and course-correct:
Get an early warning if your recent series of improvements is actually causing unintended harm and trending negatively. This allows your team to intervene and course-correct before long-term damage occurs.
Set a culture of true accountability:
Use Holdouts to bring transparency and align your teams towards one common vision, by tying feature launches directly to business health. You shift your team’s mindset from simply shipping features to actually driving business outcomes.
Start visualizing your true roadmap ROI today!
Holdouts are currently available for the Enterprise Plan (Early access) of VWO Feature Experimentation. Please reach out to your Customer Success Manager to get access.
To know more, head over to our Knowledge Base.
If you have any questions or feedback, we are all ears at support@vwo.com.
