VWO SmartStats

SmartStats reduces your testing time by half

Get more intelligent results to make smarter decisions.

What Is SmartStats?

SmartStats is our all-new Bayesian-powered statistics engine for A/B testing, which produces results much faster than before and gives you more control over your testing. You can now plan better, have a more accurate reason to end tests, and really understand how close or far apart A and B are.

Isn’t that what you really want from your optimization platform?

SmartStats is faster than Frequentist!

Compare the average duration of a campaign on a Frequentist A/B testing tool with that on Bayesian-based SmartStats.

DAILY AVERAGE NUMBER OF VISITORS
CURRENT CONVERSION RATE
EXPECTED CHANGE IN CONVERSION RATE

You will get results in:

Traditional A/B testing

Using Frequentist Statistics

108

days

New SmartStats

Using Baysian Statistics

33

days

69% FASTER!

* This is an approximate duration for an A/B test with control and one variation.
You are recommended to run a test for at least 7 days or an integer number of weeks to include weekend traffic and equal number of Mondays, Saturdays etc.

Get more intelligent results for making smarter decisions

Precise Conversion Rate range

SmartStats calculates the precise range of your true conversion rate.

More

Potential Loss

Understand what you stand to lose if you deployed A when in fact B was actually better.

More
Bayesian Reports in VWO

Probability of A beating B

Forget significance, SmartStats gives you the actual probability of A beating B.

More

Chance to beat all

Get the probability of a variation beating not just control but other variations as well.

More

Easily understand your results with recommended actions

We have a winner! v1 social proof outperformed all other variations including control by at least 12%.

Recommended action: Discard all other variations and go with the winning variation.

We have multiple Smart Decisions! v1 social proof and t1 testimonials have outperformed all others by at least 9%.

Recommended action: Choose any one of them, because their perfomance is almost the same.

We have a Smart Decision! v1 social proof outperformed all other variations including control by at least 12%.

Recommended action: Discard all other variations and go with the winning variation.

Smart decisions minimize the
potential loss

Potential loss is the lift you would have lost if you would have deployed A as the winner, when in fact B was actually better. SmartStats uses potential loss to end tests. When the potential loss of a variation drops into your comfort threshold, SmartStats suggests that it’s a smart decision to deploy the variation. With a smart decision, you stand to improve your conversion rate and will, most likely, not lose anything because the potential loss from a false positive is now minimized.

How SmartStats uses potential loss to declare Smart Decisions

Smart Decisions in VWO Smart Decisions in VWO Smart Decisions in VWO Smart Decisions in VWO Smart Decisions in VWO
  • Day 0
  • Day 2
  • Day 7
  • Day 8

All tests start without any assumptions about the conversion rate (CVR) of A and B.

As SmartStats starts recording conversions, it calculates the highly likely values for A and B’s CVR. There isn’t enough evidence to declare a smart decision on Day 2.

A still cannot be declared as the smart decision because there are still highly likely values for B beating A and so there is a high potential loss.

We now have enough evidence to declare A as the smart decision. There are barely any highly likely values for B beating A and hence a very small potential loss.

chris stucchio

Instead of trying to find the absolute truth, SmartStats looks for the better choice. You get intelligent answers quickly so that you get more time to increase your conversions and more time to test more ideas.

CHRIS STUCCHIO

Ph.D, Director of Data Science - VWO

Get greater control over your testing

Potential loss is the lift you can lose if you deployed A as the winner when in fact B was actually better. SmartStats uses potential loss to end tests. When the potential loss of a variation drops into your comfort threshold, SmartStats suggests that it’s a smart decision to deploy the variation. With a smart decision, you stand to increase your conversion rate and will, most likely, not lose anything since the potential loss from a false positive is minimized.

How SmartStats uses potential loss to declare Smart Decisions

Potential Loss in Bayesian Testing in VWO Potential Loss in Bayesian Testing in VWO Potential Loss in Bayesian Testing in VWO Potential Loss in Bayesian Testing in VWO Potential Loss in Bayesian Testing in VWO
  • Day 0
  • Day 2
  • Day 7
  • Day 8

As SmartStats becomes surer about the conversion rates of A and B, it better estimates the potential loss of chosing A. By choosing a mode, you define a threshold and once the potential loss falls under this threshold, SmartStats ends the test and declares a smart decision.

As SmartStats becomes surer about the conversion rates of A and B, it better estimates the potential loss of chosing A. By choosing a mode you define a threshold and once the potential loss falls under this threshold, SmartStats ends the test and declares a smart decision.

As SmartStats becomes surer about the conversion rates of A and B, it better estimates the potential loss of chosing A. By choosing a mode you define a threshold and once the potential loss falls under this threshold, SmartStats ends the test and declares a smart decision.

As SmartStats becomes surer about the conversion rates of A and B, it better estimates the potential loss of chosing A. By choosing a mode you define a threshold and once the potential loss falls under this threshold, SmartStats ends the test and declares a smart decision.

How to choose a suitable mode

Quick Learning

Great for finding quick trends.

When can you use it?

With this mode, you can reduce your testing time for non-critical tests when there isn’t a risk of hurting your revenue directly by deploying a false winner.

Balanced

Ideal for most tests

When can you use it?

This is the default mode and can be used for almost all tests. As the name suggests it is the best balance between testing time and minimizing potential loss.

Maximum Certainty

Best for revenue-critical tests when you want to absolutely minimize potential loss

When can you use it?

Suppose you have an eCommerce website and you want to test changes to your checkout flow. You want to be as certain as possible to minimize potential loss from deploying a false winner even if it takes a lot of time. This is the best mode for such critical tests which affect your revenue directly.

Use a better methodology with no extra effort

Peeking bias eliminated

You no longer have to worry about getting inaccurate results because of regular monitoring of your tests.

Automatic estimation of sample size

As you set up the test, SmartStats automatically estimates the number of users you need to test based on different factors.

Proactive suggestions for accurate testing

SmartStats is smart enough to detect methodology mistakes and guides you to get the most accurate results.

The ultimate guide to SmartStats

Get everything you need to know about SmartStats and Bayesian A/B testing on email.

Schedule A personalized demo

Get in touch with us for a detailed walk-through of VWO.

Request a Demo

Contact Us / Login

Product
Resources Home