We’re thrilled to introduce our new VWO’s enhanced SmartStats – Bayesian powered sequential testing engine that powers our reports. Our enhanced statistical engine gives you full control of the statistical parameters of your experiments to ensure precise, statistically robust results for confident decision-making.
The enhanced configuration options let you measure conversions, analyze reports, act fast with timely recommendations, and monitor experiment integrity.
Here’s what’s new:
1. Define the direction of improvement for your metrics
When setting up a metric in VWO, you specify the event to track. Now, you can also define whether the desired direction of improvement is upwards or downwards.
By setting the direction of improvement, you can define metrics where a lower value is more desirable, such as bounce rate, refund rate, or page load time, and use them in your campaigns. Our stats engine will interpret results based on these predefined directions. Once configured, these metrics can be saved and reused for future campaigns, streamlining your workflow.
2. Protect business-critical KPIs with guardrails
In addition to performance metrics, you can set up guardrail metrics to protect business-critical KPIs. These guardrail metrics act as a safety net, ensuring that any negative impact from experiments is quickly identified. If the value of these metrics drops below a specified threshold during a campaign, the campaign automatically pauses, and you receive a notification. This proactive measure helps prevent potential damage to your business during campaigns, ensuring that your experiments do not adversely affect essential KPIs.
3. Greater control and flexibility
Now, you can customize critical statistical parameters such as Region Of Practical Equivalence (ROPE), Minimum Detectable Effect (MDE), False Positive Rate (FPR), and statistical power. The flexibility to configure allows you to tailor experiment configurations according to your business needs, balancing accuracy and resource requirements.
- Minimum Detectable Effect (MDE) empowers you to optimize experiments on low-traffic pages. Increasing the MDE allows you to run experiments more effectively, reach conclusions faster, and conserve resources. While this approach might overlook small improvements, it significantly accelerates decision-making.
- Region of practical equivalence (ROPE): ROPE lets you define a range around the baseline where conversion changes are considered equivalent to the baseline to ignore very minute uplifts. For example, if you have a conversion rate of 20% and you define a ROPE of ±5% (relative to baseline), then any change of conversion percentage between 19% – 21% is treated as equivalent to the baseline. This allows our stats engine to prioritize meaningful improvements and disregard minor variations.
- The False Positive Rate (FPR) lets you control the chance of falsely detecting significant effects. Adjusting the FPR balances efficiency and accuracy. A higher FPR enhances detection for non-critical campaigns, while a lower FPR ensures precision for critical campaigns.
- Statistical Power is the probability of accurately detecting a true effect. The configuration allows you to set a higher power for critical metrics to identify small improvements and a lower power for less critical metrics to save resources while ensuring reliability.
For example, when testing new messaging, you can lower the MDE to get quicker results on low-traffic pages, enabling faster experiment conclusions. Similarly, you can configure higher statistical power for critical experiments, such as pricing campaigns, and lower it for non-critical campaigns, such as minor design tweaks.
You can refer to this set of knowledge base articles to understand more about statistical configuration for your experiments.
4. Get smarter, accurate decisions faster
Our stats engine recommends the analytically best decision that answers your business question—“Is my variation better than my baseline?” Our engine declares a variation as the better variation ready to be deployed if its probability of improvement reaches 95% and recommends disabling it if its probability falls below 5%. Our engine even internally corrects for common statistical biases.
When a variation is statistically unlikely to perform better than the baseline, the engine recommends disabling it from the experiment. In such cases, you can disable it to prevent conversion losses to an underperforming variation.
A separate testing mode called ‘Non-inferiority tests’ allows you to save visitors by testing on the relaxed condition of “improvement or equivalence.” Suppose you launch a new homepage design and only want to test that conversion rates should be equivalent or better (but not worse). You can reach significance faster in such campaigns.
5. Built-in error-handling
Our enhanced stats engine can now handle common statistical errors that can affect the validity and reliability of data analysis, such as “Peeking problem” and “Multiple comparison problem,” to give you reports with real-time data that are valid whenever you look at them.
Sequential Testing
Sequential testing correction solves the “peeking” problem, where calculating statistical significance multiple times during a test increases the chance of false positives. VWO adjusts the probability of improvement based on the test’s maximum sample size, allowing you to monitor results continuously and make timely decisions without compromising accuracy.
Bonferroni Correction
In tests with multiple variations, error rates increase when comparing each variation to the control. Traditional Bonferroni correction adjusts statistical significance thresholds to control error rates. VWO corrects this bias without lowering the statistical significance threshold. Instead, our engine widens the confidence intervals, allowing for accurate interpretation without adjusting decision boundaries.
6. Fixed horizon tests
Fixed horizon test calculates results only after collecting the entire sample size. Conclusions are made after the campaign’s pre-set duration ends, making it ideal for enterprises ready to wait until the end of sample collection for results and don’t want to take the risk of peeking errors.
7. Consolidated checks on experiment health
The upgraded statistical engine continuously tests for conditions that might compromise an experiment’s integrity. It monitors parameters such as data tracking, minimum 7-day runtime, and experimentation conduct.
Any breaches in these checks indicate potential flaws in the campaign. If the engine detects any issues with these vitals, it will notify you so you can inspect and take corrective action if necessary. You can read more about experiment vitals in this article.
8. Improved visuals with graphical representation
You can understand their performance at a glance thanks to clear graphical insights for each variation. Switch to in-depth data review mode for a detailed view showing expected improvement, improvement intervals, and probability of improvement for each variation to thoroughly understand your experiments’ progress.
Ready to dive in?
Step into the future of experimentation with VWO’s enhanced reports. Enhanced reports will now be available in all VWO Testing and VWO Feature Management plans. Log in to your VWO account today to experience the power of our enhanced reports! If you are an existing customer, you can choose between the classic stats engine and the enhanced stats engine for your new campaigns until this year. You cannot switch the stats engine for a running campaign. Also, your existing campaigns will remain unaffected by this release.
If this got you excited, we’re just getting started. Stay tuned for upcoming exciting updates!
For more information or assistance, contact support@vwo.com. Our support team is here to help you every step of the way.
