Chasing Just Win Rates Incentivizes Your Team to Test Low-Reward Ideas

Our CRO Perspectives series captures lessons from practitioners and industry leaders who are reshaping experimentation in real time. 

In this 25th post, we explore what it takes to build experimentation programs that can scale across products, teams, and increasingly AI-driven workflows without losing sight of the people making the decisions. 

Leader: Apurva Sandbhor

Role: Apurva Sandbhor, Manager – Platform & Product Experimentation, The Home Depot

Location: Atlanta, United States

Speaks about: Experimentation Strategy • Product Experimentation • AI in Experimentation • Experimentation Governance • Testing at Scale

Why should you read this interview?

Most conversations about experimentation focus on tools, tactics, or test outcomes. Apurva shifts the focus to what makes experimentation sustainable at scale: the systems, operating models, and leadership principles that help teams move faster without sacrificing alignment or decision quality.

In this interview, she shares practical frameworks for scaling experimentation across large teams, reducing operational friction through automation, preparing for server-side experimentation, measuring success beyond win rates, and understanding where AI can accelerate execution without replacing human judgment.

Whether you’re scaling an experimentation program, earning executive buy-in, or evolving your testing strategy, this conversation offers practical lessons you can apply.

Scaling an experimentation team without compounding friction

Scaling a team without scaling your platform architecture is simply compounding friction. When organizations try to scale experimentation, the knee-jerk reaction is often to throw headcount at the problem. But if you don’t have the right structural and cultural frameworks in place, a larger team just means more siloed execution, metric discrepancies, and operational noise. To keep an enterprise organization aligned and moving at velocity, leadership requires a shift from a reactive support mindset to a platform-first strategy.

To keep an enterprise organization aligned and moving in the same direction, here is how leaders can approach work distribution, alignment, and scaling frameworks:

1. Work distribution: The “Dual-Lane” framework

To prevent a centralized experimentation team from getting bogged down by reactive, transactional testing requests, you must decouple day-to-day enablement from core innovation. This can be handled through two distinct mechanisms:

• Specialized pods – structured cross-functional team of software engineers, data engineers, QA specialists and data analysts into hyper-focused pods. One pod owns platform modernization (infrastructure, automated pipelines, AI decisioning engines), while the other focuses on process optimization (governance, scaling workflows, and uncovering actionable insights).

• The Dual-Lane framework: Every single team member operates across two distinct tracks. Lane 1 is Foundational Execution—providing the guardrails, tools, and code architectures that empower product & marketing squads to self-serve their experiments safely. Lane 2 is Platform Advancement—stretch initiatives where engineers & analysts explicitly own long-term features that uplevel core capabilities. This ensures nobody gets burned out by repetitive execution and everyone is actively building the future of the platform.

2. Rallying people: Anchor on “Learning Value,” not vanity metrics

It is incredibly easy for an organization to lose its way if it is rallying around the wrong North Star. If your core OKRs focus heavily on raw test volume or “win rates,” you will inadvertently incentivize your team to test low-risk, low-reward ideas just to hit a number.

To keep cross-functional squads moving in the same direction, it is vital to shift the entire enterprise narrative away from intuitive guessing and volume mandates, re-anchoring goals around learning value and decision integrity. A culture is built that cleanly distinguishes between:

• Validation: Running tests to mitigate risk and safeguard existing revenue.
• Exploration: Giving squads the psychological and architectural safety to test bold, disruptive hypotheses.

When your engineers, data scientists, and business stakeholders understand that the goal isn’t just to launch a test, but to securely unlock a high-integrity business insight that shapes corporate strategy, alignment follows naturally.

3. What leaders must keep in mind before expanding headcount

Before scaling your team, leaders must:

• Audit the infrastructure latency: Fix the systemic pipeline bottlenecks first. Invest heavily in automation—from intake to data ingestion to automated report generation.

• Offload transactional analysis: Empower a separate business analytics unit or decentralized product squads to run the day-to-day experiments, allowing your core team to remain focused strictly on platform innovation and scalability.

• Hire for complementary rigor: Don’t just hire for volume. Build a cross-functional matrix that explicitly bridges execution, data science rigor, and platform development within a single group.

Scale your platform’s capabilities first, and your velocity will naturally increase without needing to linearly scale your human capital.

Before you write a job description to expand your headcount, ask yourself one critical question: “Are we hiring people to manually run more tests, or are we hiring people to build a platform that acts as a force multiplier?” If your current operating model relies on a centralized team manually building, QA’ing, and analyzing every single test, expanding headcount is just a band-aid. You will eventually hit a ceiling.

Speeding up testing without compromising data integrity

Accelerating velocity without safeguarding data integrity is an existential risk to enterprise product strategy. True velocity is not achieved by rushing data interpretation; it is achieved by automating operational and statistical guardrails directly into the platform architecture.

Re-architecting the enterprise platform has cut down our end-to-end testing lifecycle by half, while simultaneously increasing statistical reliability. This required a focus on three targeted architectural and operational methods:

1. Embedded statistical guardrails and automated checks

The greatest drain on experimentation velocity is time spent on poorly designed tests or false positives that require re-testing. This friction is eliminated by integrating a proprietary statistical engine directly into the ingestion pipeline:

• Automated sample size & power analysis: Before a single line of test code is deployed, the platform automatically executes power analyses based on historical baseline data. This prevents teams from running underpowered tests that waste time, or over-extended tests that waste traffic and delay decisions.

• Early verifications: The platform automatically verifies data pre-requisites and sample ratio mismatch (SRM) checks during the early stages of a test, flag-raising data anomalies in real-time rather than waiting for post-test analysis.

2. Full operating model automation (End-to-end pipelines)

Velocity suffers when manual handoffs persist between product squads, QA, and data analysts. Replacing a fragmented operational workflow with a seamless, automated platform ecosystem streamlines execution across clear, programmatic stages:

• Automated setup and QA: Implementing self-serve intake and deployment infrastructure automates the generation of experiment code and progressive feature-flag routing. This allows cross-functional teams to build, QA, and launch tests autonomously without waiting for centralized support.

• Automated reporting pipelines: Automating the entire backend pipeline ensures that the moment an experiment concludes, data processing engines automatically handle data retrieval, apply core statistical frameworks, and generate a standardized, executive-ready results report. Experimentation SMEs can add actionable insights and recommendations to this report, elevating the learnings and enforcing the iterative process of experimenting. 

3. Decoupling deployment from activation via modernized architecture

To facilitate rapid iteration without risking user experience, infrastructure must be modernized to support a hybrid client-side and server-side model:

• Progressive feature delivery: By utilizing advanced feature flagging, engineering teams can deploy code to production silently, allowing product teams to activate, scale, or instantly kill experiment traffic via platform controls.

• Minimizing performance latency: Shifting to server-side experimentation ensures that testing complex backend logic or macro feature changes does not degrade digital performance, protecting the user experience while unlocking fast, high-integrity insights.

Velocity and quality are not a trade-off; they are mutually inclusive. When experimentation is treated not as a series of manual analytical projects, but as a product-driven, automated infrastructure system, an enterprise gains the guardrails to move fast without sacrificing precision.

What must be in place before moving to server-side experimentation

Graduating to server-side experimentation requires a cultural and structural evolution. Before investing in advanced infrastructure, an enterprise must possess a mature culture of data-driven decisioning and an appetite for scaled testing to justify the ROI. This demands a robust change management plan that prioritizes training, education, and clear stakeholder incentives to accelerate platform adoption.

The foundations

• A unified statistical engine must already govern client-side testing. If data integrity and baseline metrics are not trusted at the UI level, server-side execution will only multiply data noise.
• Core feature flagging capabilities must be stable, cleanly separating code deployment from release activation.

The overlooked steps

• Businesses frequently fail to establish operational governance for stale flags. Without strict, automated cleanup workflows, progressive rollouts rapidly create massive technical debt.

• Shifting from client-side visual tweaks to deep backend logic changes requires engineering and product management to re-align. If product squads lack the technical fluency to design server-side variables, platform adoption stalls.

What must be in place before moving to server-side experimentation

I truly believe that experimentation is a continuous engine, not a linear project. Closing the loop between a concluded test and the next iteration requires a systematic pipeline of compounding insights rather than a “launch-and-forget” approach.

The insight-to-hypothesis loop

1. Post-test dissection – Verifies metric integrity and isolates data anomalies immediately upon test conclusion.
2. Segment deep-dives: Slice data by key user cohorts to uncover hidden behavioral patterns rather than relying on flat, aggregate results.
3. Knowledge hub: Index all findings—wins, losses, and neutrals—into a centralized repository to prevent duplicative testing and scale institutional knowledge.
4. Hypothesis refinement: The exact variance between expected user behavior and actual data becomes the explicit, data-validated baseline for the next hypothesis.

To run this pipeline smoothly without pre-existing infrastructure, a new product team requires three non-negotiable elements:

• The core cross-functional trio:

• Product leader: Translates business friction into prioritized, testable ideas.
• Data analyst & SME: Establishes measurement frameworks and guards statistical integrity.
• Implementation engineer: Ensures clean code execution and progressive feature delivery.

• Standardized hypothesis: Every backlog item must adhere to a strict template: “Based on [Prior Data], changing [Variable] for [Segment] will impact [KPI].” This removes political bias and executive opinion from the testing queue.
  Prioritizing the tests based on impact, LOE and business strategy is vital.

• Metric alignment: Align Finance, Product, and Engineering on core data definitions before launching the first test to eliminate post-test debates over metric validity.

Shift the cultural focus from “win rates” to “learning velocity.” Measuring teams by the quality of insights unlocked eliminates risk-averse testing and keeps the optimization loop running smoothly.

When an unexpected result reshaped a feature launch

During a high-visibility checkout optimization rollout, an algorithmic recommendation engine designed to cross-sell accessories unexpectedly caused an aggregate drop in cart conversion rates.

Instead of abandoning the initiative, further analyses and segment deep-dives were conducted. The data revealed a critical behavioral nuance: while the algorithm performed exceptionally well on product discovery pages, injecting multiple individual product choices directly into the high-intent checkout path triggered severe decision paralysis.

This behavioral insight led to a refined hypothesis: replacing the multi-choice layout with a single bundle would eliminate cognitive friction while retaining cross-sell value.

The follow-up test validated this, converting the drop into a net revenue lift and redefining the final feature release.

Deciding whether to double down, iterate, or roll back

Once a feature is live, performance analysis shifts to automated executive scorecards paired with real-time system health monitors. Automated anomaly alerts immediately flag critical shifts in operational latency or metric guardrails for instant rollback decisions.

Deciding to iterate or double down requires looking past flat, surface-level reports. Instead, automated cohort segmentation and long-term downstream tracking are leveraged to catch delayed user friction and measure true business impact.

Catching false positives before they turn into costly decisions

Data anomalies are caught early by embedding sample ratio mismatch (SRM) checks, customer bucketing checks during the early stages of test and power analyses directly into the ingestion pipeline. When conflicts occur, decisions follow a strict risk-reward matrix. If a test triggers a high metric volatility that goes beyond the pre-defined threshold, it is paused immediately. The business impact is weighed by balancing potential revenue erosion against wasted engineering cycles. If data integrity is compromised, the test is scrapped; if it is an infrastructure anomaly, it is rerun. Protecting customer trust always supersedes shipping features.

Keeping executives engaged when a test fails

Executives stay engaged when the narrative shifts from “winning” to protection and safeguarding innovation. The most impactful slide shown to leadership is a “Strategic Risk Avoidance” slide, which calculates the estimated $ revenue loss prevented by stopping a heavily backed but flawed feature from going live. 

Inconclusive tests are framed using learnings and actionable insights, showcasing how these have informed the next backlog item or iteration. Demonstrating that a “loss” saved capital keeps executives bought into experimentation as an ironclad insurance policy.

What AI can never replace in experimentation leadership

As AI commoditizes the execution layer–building test variants, automating data pulls and analyses, operational excellence, enforcing guardrails, the role of a product leader or CRO evolves from tactical manager to strategist.

Human judgment remains irreplaceable in three areas: defining the ethical and strategic boundaries of testing, translating complex user psychology into novel hypotheses, and aligning cross-functional stakeholders around a long-term vision. AI can optimize the path, but humans must choose the mountain. Leadership means curating the portfolio of risk, not managing the tools.

Why win rates are the wrong metric for a healthy program

I have seen leaders dictate that a healthy experimentation program is the one that launches thousands of tests and boasts a high percentage of positive results. The defining realization of a strategic platform leader is the exact opposite. When organizations over-index on raw test volume or win rates as core OKRs, it creates an insidious cultural side effect: it incentivizes teams to test low-risk, low-reward visual tweaks that guarantee minor wins but completely starve the company of true innovation.

As Harvard Business School Professor Stefan Thomke notes in Experimentation Works:

“For every online experiment that succeeds, nearly 10 don’t… Overemphasizing the importance of successful experiments may inadvertently encourage employees to focus on familiar solutions, or those that they already know will work, and avoid testing ideas that they fear might fail.”

To evaluate a technical ecosystem’s true business leverage, leadership must shift the paradigm toward metrics that map directly to shareholder value:

• Strategic risk avoidance rate: In Trustworthy Online Controlled Experiments, industry pioneer Ron Kohavi notes that 60% to 90% of ideas fail to improve the metrics they were designed to advance. Therefore, the primary financial value of an experimentation platform is protection—measuring the millions of dollars safeguarded by stopping flawed, highly anticipated product deployments from ever hitting production.

• Speed to learn: The critical operational engineering metric is minimizing the cycle time between hypothesis ingestion and automated report delivery. True velocity means shrinking lifecycle friction allowing product squads to iterate faster based on empirical evidence.

• The enterprise learning rate: A healthy program measures the velocity at which “failed” or neutral experiments are converted into actionable user-behavior insights. If a team can turn a rejected hypothesis into a refined, high-integrity backlog item within days, the infrastructure is actively driving innovation. 

Volume is an input; decision integrity and risk mitigation are the ultimate outcomes. The most valuable program is not the one running the most tests—it is the one that uncovers business truths the fastest while protecting the enterprise from costly strategic missteps.

Conclusion

The strongest message running through Apurva’s perspective is that successful experimentation is built on the systems, operating models, and mindset that help teams scale, learn continuously, and make better decisions.

From distributing work across large teams and automating experimentation workflows to redefining success around learning value and risk reduction, she makes the case for treating experimentation as a long-term capability rather than a series of isolated tests.

Putting these principles into practice requires the right foundation. VWO AB Tasty brings together automated guardrails, AI-powered insights, enterprise-grade governance, and more to help teams scale experimentation. Book your personalized demo today.