How to Find Hidden Winners in Your A/B Tests
UC Berkeley was once sued for bias against women applying to their graduate programs. Their fall admission figures showed a difference of 25% which couldn’t be just attributed to chance. They successfully defended against this charge with a drill down to department-level segmentation which proved that in fact women had a higher admission rate for most departments; it was just that majority of the female applicants had applied for competitive departments which already had a low admission rate for both genders hence skewing the overall data. Welcome to Simpson’s Paradox and how you can beat it using Post-Result Segmentation to find trends for a certain segment of traffic which could be buried under a mountain of data.
Post Result Segmentation
Post Result Segmentation in VWO is an enterprise feature which lets you slice and dice your reports to smaller, specific and targeted segments. This lets you find out the most profitable segments and double down on them. You can apply complex segmentation parameters to find the hidden winners in your test. Setting segmentation parameters before starting a test isn’t always very easy. You can’t anticipate and pre-specify all the segments accurately. Luckily, VWO’s Post Result Segmentation feature doesn’t need you to. In tests with the feature enabled, all you gotta do is load up the report and get slicing.
How Do You Segment?
Frankly, the parameters for segmentation could be anything. They can range from gender of the visitor to the demographic they are from or the operating system they use. Avinash Kaushik of occam’s razor broadly classifies the segments in three different categories:
- By Source: Separate your organic traffic from your paid traffic, traffic that comes from social sites and referrals.
- By Behavior: New vs Returning visitors, visitors who are from the US vs visitors from Europe. Different visitors have different intents to come to your website, it is important that you realize what they want and make sure they get it!
- By Outcome: Separate people out by the the products they purchase, the size of purchase, or people who filled a form vs who made a purchase.
Mirela Belkoski is the UX manager of V&D. V&D is a leading ecommerce store in Europe. She says she starts off her analysis with a quick check of segmenting visitors on multiple devices, new vs returning visitors and sources of traffic (paid vs organic vs social). If the test runs for specific categories, she also does custom segmentation to find if there are any differences within the categories. Mirela also shares with us, a test experience where V&D was testing USPs (Unique Selling Points) on the site to find out what ‘seduces’ their customers the most. The USPs would say,’free delivery above €75’, ‘free pick-up in store‘ or ‘free returns in store’. On the surface it didn’t seem like anything was happening, but diving deep they found out that the conversions for women’s fashion had significantly gone up whereas the men’s fashion was showing a decrease.
Segmentation Does Wonders Once You Dive Deep!
An issue a lot of websites face is that there are too many goals attached to them; visitors might be there to subscribe to your world class blog; to get a sales demo; perhaps to just get the product’s information. It doesn’t make sense to analyze every one of them under one umbrella. Segmenting gives tremendous amount of insight into what an individual type of customer expects from your site. Becky Simanowski is the CRO expert of SAS, which is a leading Business intelligence and Analytics software. Becky says setting up a simple new vs returning filter gave her a ton of insight into the type of information different segments of visitors expect from their website. She then targeted the changes to specific segments leading to happy conversion rates.
Here are the Experts’ Most Favorite Segmentation Parameters
- Device type: Mobile vs Desktop
- Location: Specific Country vs Global
- Traffic Sources: Paid vs Organic vs Social
- Users: Engaged (Active) vs, well, not so engaged.
Now then, load up your reports and start slicing and dicing! We all would love to hear your experiences with Post Result Segmentation.