Why Google Content Experiments might not fulfil your testing needs
Google Website Optimizer is morphing into the watered down Google Content Experiments, which is now available to everyone. A lot has already been written about how this will affect the Conversion Rate Optimization industry. However, resources on what this means for business owners are few and far in between. Therefore, we wrote this guide on what GWO closing down means for your online business and what steps you must take to remain ahead of the curve.
1) Integration with Google Analytics – First and foremost, Content Experiments is now a part of Google Analytics and this means better integration between the two. You only need one Google Analytics UA number unlike previously, where two different UA numbers were required. This also means less code to be placed on the page because you’re inserting only one tag now. Creating experiments are still fairly easy but you’ll have to use Google Analytics goals whereas in Google Website Optimizer, you had the ability to consider any page or button a conversion goal.
You’re also limited to creating only Split URL tests, i.e. your visitors will be split between two different pages whereas other tools offer A/B testing on the same page (visitors are split between different versions of the same page).
2) Your experiments will be terminated after 3 months – All content experiments currently have a maximum life of 3 months. If you want to see data after that, you’ll have to create a new experiment and combine the data between the two. Usually, small improvements take a lot of time to achieve statistical significance while large improvements happen fairly quickly. If any of your experiments used to take longer than 3 months, you’ll have to learn to create bigger changes that show bigger results.
3) Experiments have to run for 2 weeks before Content Experiments reports statistical significance – This is a constraint if you have a large number of visitors to your website and statistical confidence can be calculated in a timeframe lesser than 2 weeks. In such a scenario, you could end up losing sales as part of your visitors will be shown a low performing variation for a period of time that is longer than required. However, test results will be shown live.
4) Advanced Segments will be available to slice & dice your data – All Advance Segments are available, meaning that you can now very finely slice & dice your data and draw better insights about your visitors’ behavior. This is a very useful addition to GCE that will enable you to finetune your funnel for different customer segments.
5) You cannot stop low performing variations before the test is over – In a running test, you will sometimes see a variation that is obviously not doing well and turn it off. This is especially so in large eCommerce websites where rogue unstoppable variations could result in a significant loss of revenues. While GCE won’t have a way to stop low performing variations during a running test, it will try to dynamically shift traffic away from them.
6) You can have maximum 5 variations per test and not more than 12 active tests – These shouldn’t be a hindrance for most small businesses but the same cannot be said for larger eCommerce websites. Websites with a lot of traffic benefit greatly from regular testing on different steps of the funnel and limiting the number of tests will prove to be a pain point.
7) You can now use URL variables and Regular Expressions – You can run tests across domains and include / exclude URLs using regular expressions. This will really help to match the tests to a path as your visitor will move from page to page. REGEX and URL variables are also helpful when you want to exclude certain IPs from a test, for example, your own company IPs so that your actions do not count as goals and skew the results.
8) No multivariate testing – This is a very important feature that Google inexplicably decided to leave out. Multivariate testing enables a quicker understanding of which changes work and which don’t. The absence of this feature is a considerable drawback for large eCommerce websites who have the visitor flow to achieve statistical significance quickly.
9) Email notifications are now available – Useful feature because you won’t have to monitor your tests constantly – set up the notifications and let Google tell you what’s happening when.
10) Revenue goals are missing – Another important functionality that Google should have included in GCE is the ability to track eCommerce transactions as goals. Currently, only URL and events based goals can be tracked so you’ll have to use a thank-you page to achieve the desired outcome. It’s not a crippling feature, but is still a “hack”.
11) Different code snippets for different experiments – What they should be aiming for is one single code snippet that allows you to run multiple tests on your page. For example, Visual Website Optimizer requires you to place the code snippet only once on a page and then, any number of tests can be run on the page simultaneously. For GCE, each Experiment has its own code snippet and you can have only one running at a time. This has two implications
- You can’t run more than one test on a step of the funnel meaning it takes longer for you to optimize your website and see the result;
- If you want to run a second test after 2 weeks, you’ll have to dig in and replace the code snippet.
In conclusion, Google Content Experiments is a tool most suitable for those just starting out with A/B testing and conversion rate optimization. It has all that any beginner would require to understand how CRO works and what it can do for an online business.
However, if you test regularly and have a website with a large number of daily or monthly visitors, you’d be well advised to go for a full fledged A/B split and Multivariate testing alternative like Visual Website Optimizer. GCE is still evolving and we are sure newer features will be rolled out in the coming days as the team behind it collects user feedback, but until then your business will need a more robust package that handles all your testing needs effortlessly.