Tutorial: managing A/B testing efforts in a (big) team
VWO is used by all sorts of organizations, from one person team to multiple product departments in a large organization. When multiple people are working on multiple different A/B (or multivariate) tests, managing them properly becomes a requirement. Without proper management of your A/B testing efforts, you may run into following issues:
- Duplication of effort: different members of your team may run similar tests on same page
- Mistakes and errors: a team member may edit, pause or stop a test which shouldn’t have been changed
- No central repository of knowledge: similar to first point, but without proper sharing of knowledge your team may not be able to apply results from one test to generate ideas for future A/B tests
There are ways to overcome these issues and, in this post, I discuss the same.
Best practices to manage A/B testing
An organization may have its own internal procedures and protocols for A/B testing, but at the least we suggest having a shared document or spreadsheet to keep tab on:
- Which pages are currently being tested and which A/B tests are in planning or design phase
- Who all in the team is responsible for these A/B tests
- Which elements are changed in these tests (buttons, headline, etc.)
- Once test is over, what is the conclusion or result from the test (e.g. green button outperformed control by 25%)
This document will help organize your efforts and you will become better and more efficient at A/B testing as your repository of results grows bigger. Of course, there must be an individual who is responsible for maintaining this document and who oversees all testing efforts in a team/organization. We have seen that without an owner and champion of A/B testing in a team, such efforts quickly fizzle out.
VWO features to help you in management
In addition to this shared document, you can also utilize multiple VWO features to organize even more efficiently. Sample some of the features available and how they should be used:
Permission based logins – This is a feature wherein you can create multiple logins for different team members. That is, if you have a large team, different team members can have their own login IDs (you don’t have to share common username/password). You can also assign different permission levels (Browse, Design, Publish and Admin) to team members according to their job role. For example, a team member who has design permissions can create a test, but s/he won’t be able to start or stop the test.
Sub accounts – This is a feature wherein you can create different accounts for different teams. Advantage of creating a sub accounts is that they will be unique for individual teams and so team members will only see their own tests (they won’t see tests from other teams). This helps to organize A/B testing efforts of different teams.
Test Name – To state the obvious, your organization can come up with a mutually agreed protocol to name tests. We have seen that organizations usually name their tests with this protocol: PAGE DESCRIPTION + WHAT IS BEING TESTED.
Notes – In VWO, you can enter notes for the test to record various points about it:
- Your hypothesis for the test (what do you expect will happen)
- Changes made in different variations
- Which team members are involved in this test
- After the test has been stopped, what were the results and lessons
We recommend extensive usage of Notes feature because many a times you check previous tests and forget the lessons / results from the test. Briefly summarizing this information in the Notes section will help preserve this information in the organization for everyone’s advantage.
Labels – Just like Labels or Folders in email programs, in VWO you can create different labels and tag tests with them. Examples of labels: homepage_test, button_test, winning_variation_found, inconclusive_test, surprising_results, etc. You can, of course, apply multiple labels to the same test. When all tests are listed in your account, labels will help you organize and filter your tests. For example, if you want to see all tests that didn’t produce any results, simply click on inconclusive_test label and see all relevant tests.
Notifications – Administrator of an account can enable Notifications for his/her account. This feature will send an instant email to the administrator upon any activity in the account. For example, if a team member starts or stop the test, administrator will get an email. This allows administrator to cross-check whether this action is something that was intended or is a mistake or miscommunication. Thanks to this email, administrator can take necessary swift actions or make necessary enquiries.
I hope you liked this post about managing A/B testing in a team environment. Do let me know if there is a particular practice that you use in your organization which others here can benefit from.