Topics to Discuss in Experimentation

In 597 BC, the king of Babylon, King Nebuchadnezzar gave an order to the overseer of his courts, Ashpenaz. He wanted some of the children captured from Jerusalem to be educated in the Babylonian tradition so that they could serve in the courts. He demanded that the children should be healthy, wise, cunning, and capable. Throughout their education, the children would be fed a diet of royal meat and wine. But as it happened, Ashpenaz had a problem. One of his favourites, a boy named Daniel, refused to touch the meat. It was against his faith.

Daniel demanded a vegetarian diet for him and his friends, but Ashpenaz refused. He was concerned that if the children looked weak after a few days on the vegetarian diet, it might cost Ashpenaz his head. Daniel then suggested creating two groups similar in all other regards, but one would be fed the royal diet and the other would be fed the vegetarian diet. He said that after ten days of this diet, he could compare and see if the two groups looked different. As the story goes, those on the vegetarian diet looked significantly healthier after the experiment, and Daniel was given a favoured place in the court for his intelligence.

The biblical story of Daniel is the earliest known example in history of a controlled experiment. Believe it or not, 2500 years later that is what A/B testing is all about. Science and experimentation have a long-drawn history, a furiously debatable philosophical foundation, and deep implications on the culture and practices of A/B testing. In this blog post, I will introduce the rough outline of this blog and the topics I will be writing about.

Broader Subsections of this Blog

Reflecting on my learnings on experimentation and the ideas that are relevant to the modern experimenter, I have outlined eight topics of interest that will be explored at length in this blog. 

1. History and Philosophy of Experimentation: Throughout history in its relentless pursuit of truth, science has often found itself on the overlap of philosophy. As has always been the case, history and philosophy have had profound implications on how to proceed into the future, which questions to ask, and what to expect. Understanding this context helps in building an intuition into why experimentation is necessary and what is the fundamental inquiry of science. This section will be devoted to exploring all these aspects and convincing modern experimenters that they are the scientists of the 21st century. 
   
2. Statistics of Experimentation: Statistics is inseparable from experimentation because the world we live in is random and statistics is the tool to deal with the same. Experimentation relies on statistical significance and the topic is much deeper and much more complex than what it looks from the outside. In this section, I intend to share my research supported with simulational results taking the reader through a journey of developing an intuition about randomness. The section will also explain in detail the statistical engine of VWO, the problems it solves, and those that we are still in the process of solving.
   
3. Culture of Experimentation: The biggest barrier to a successful experimentation effort in modern tech companies is neither the tool nor the statistics but rather the culture of trusting data to guide your work. The culture of experimentation is a deeply philosophical and social topic that deals with building the right perspective and stimulating the right conversations around A/B testing. I intend to take the reader through various ideas such as tackling the Hippo, developing a hyperrealist approach to life, and accepting failures. Experimentation demands modern leaders to actively inculcate the necessary cultural ideas first into themselves and then into their companies. 
   
4. Metrics in the Digital Era: The invention of computers and the exponential growth of data has given modern experimenters the privilege to lead with the help of meaningful metrics. Experimentation efforts rely on the choice of metrics and businesses in the 21st century are viewed from the lens of effective metrics. If the business is a ship, metrics are the compass that helps the captain guide the ship toward the north star. But metrics as a topic is very complex and has many nuances to it that are often not talked about in detail. This section will explore the various properties of metrics, the different types of metrics, and the necessary science that governs how metrics should be chosen and improved.
   
5. Practical Know-Hows in A/B Testing: While A/B testing is still a novel practice in modern businesses, a lot has been learned about the effective ways of A/B testing that can be leveraged to avoid initial failures. Experimenters across the globe have shared their experiences and mistakes in the form of stories and concepts. This section will explore the practical aspects of A/B testing and the things that the modern experimenter should know. I will be writing about concepts such as interaction effects, weekend-weekday effects, successful ideas to try, and triggered tests. 
   
6. Components of an Experimentation Platform: With the popularization of A/B testing, a new industry has propped up into the market that automates all the technical details of experimentation making it simple for modern experimenters to run A/B tests. An experimentation platform in that regard is crucial to the success of experimentation. As we are learning more and more about the needs of modern experimenters, we are also developing a vision of the components an experimentation platform needs. In this section, I will be talking at length about the nuances of these components, how they help the modern experimenter, and where this newfound industry needs to progress to fulfil the needs of its customers.
   
7. Advanced Algorithms in CRO and Experimentation: Algorithms have taken over the world and A/B testing is just the mere tip of the iceberg when it comes to data-driven decision-making. Once a business gets started on their experimentation journey, they realize there is so much more to do in conversion rate optimization. A business can personalize their websites for different segments, recommend the optimal items to the optimal customers, slice their data to learn customer behaviour in-depth, and derive trustworthy heuristics by meta-analyzing multiple experiments. This section will be focused on revealing what more is possible with data beyond vanilla A/B testing.
   
8. Causal Inference: Causal Inference is a newly emerging field that ties together many concepts of science and gives a perspective into what the future looks like when machines will be able to learn cause and effect relationships on their own with data. Many new optimizations in A/B testing have their roots in the concepts of causal inference and many unexpected effects in experimentation can be explained with these concepts. This will be the final section I chose to write about that will place experimentation in the futuristic context and will help modern experimenters realize where they are headed.

The Way Ahead

As mentioned in previous blogs, I aim to build VWO Stats Blog into a holistic source of developing a broad perspective on experimentation for the modern experimenter. This is the rough structure that I am starting out with and I expect this to evolve further as my own readings and understanding of experimentation improves. Out of all the topics mentioned above, statistics will remain the center of focus and I aim to pick up the other topics one by one while writing regularly on statistics. I request the reader to join in, contribute to my knowledge, correct me if I am wrong, and make this a collaborative learning experience.

Thank you sincerely for your time and attention.