Introduction: Why read this blog?

One day in 2012, a Bing employee suggested the idea of lengthening ad headlines by adding a few words from the first line on the page. The idea languished on the product roadmap for months before an engineer just implemented it one day because it was so easy to do. He then launched an experiment on a small proportion of random visitors.

Bing servers raised a high-revenue alert and the team got together to debug the error in the pipeline. Some thought it was a logging issue, others thought it was motivated by sudden externality. End of the day, nothing was found. It was just a simple experiment that had boosted Bing’s revenue metrics by 12%, all in all by $100 million/year. All because someone tested an uncertain idea.

In the past few decades with the emergence of a digital world, we had an explosion in the space of possibilities. In the business world, the possibilities have exploded in what you can build, how you can drive it, and the various niches of success you can exploit. The truth lies with the customer and our perception might be remotely far from what really works. The only way to navigate through this mesh of possibilities is to wear our scientist’s hat, be ready to fail, and test all our hypotheses.

A/B Testing is the modern-day digital adaptation of scientific experiments that we have been doing ever since the discovery of science. In the last 100 years, we have built the statistics to experiment in an increasingly uncertain world with the invention of randomized controlled trials. In the past 20 years, we have amassed an unprecedented capability to measure and record all kinds of data.

Today, we are sitting on some of the most accessible ways to experiment in our work. This blog is an attempt by us at VWO to share our learnings on the topic with the modern-day experimenter.

The Three Pillars

The modern-day experimenter lies on a broad spectrum ranging from a digital marketer to a product manager to an entrepreneur. While the nuances of implementation might be different for different experimenters, the broader intuition behind the work is the same.

VWO Stats Blog aims to unravel this broader intuition and revolves around three core themes:

1. Experimentation: While A/B testing is an approach to optimize your website, the broader context of experimentation is an ideology. Experimentation involves innumerable types of complexities, assumptions, patterns, and errors that most of us are unaware of. In my opinion, delving into the science, history, and philosophy of experimentation is the way towards becoming digital scientists. In that regard, my aim with this blog is to share the most interesting stories and concepts that will allow you to appreciate the true depth of experimentation as a field.
   
2. Statistics: If experimentation is the protagonist of this blog, then randomness is the antagonist. Experimentation evolved to be such a deep subject only because randomness perpetually finds creative ways to defeat your intentions. Statistics is the mathematical study of randomness and the cornerstone of successful experimentation. Statistics gives you the right intuition for identifying the creative ruses of randomness and a loaded toolbox to dissect these problems. My aim with this blog is to take the reader through statistics in a way that they are able to intuitively grasp the core concepts behind complex statistics.
   
3. Being Data Driven: They say data is the oil of the 21st century and being data-driven has much more to it than what it looks from the outside. Asking the right questions, gathering the right metrics, and then understanding data is the broader culture that gives space for experimentation to grow. With the right stories and cartoons, I want to share a deeper mindset of knowing when to trust the data and when to trust your intuition.

VWO Stats Blog aims to blend in these three topics in coherent prose and interesting stories in the form of short and digestible blog posts.

The Three Key Takeaways

When I try to think about the audience I want to cater to, I can think of three distinct takeaways that the audience can derive out of my writings.

1. A chance to experiment with making experimentation a way of life: When you start to appreciate experimentation as a way of life, you don’t just A/B test banners on your website, you even start testing different ways of telling your anecdotes. In my opinion, the deeper context of experimentation makes us realize that the fastest way to move through an uncertain space is by contradicting yourself and in the cheapest way possible. I want to motivate all readers to adopt the experimental perspective to life.
   
2. A refined understanding of randomness in an uncertain world: It is likely that as you start to get entangled more and more in data, you will be frequently greeted by the ugly face of randomness messing up the inferences you make. Paradoxes and biases are the daily dealings of numbers and randomness. I want to share my personal journey in statistics to leave the reader with an intuitive map of navigating through the problems of randomness.
   
3. A repository of jokes and anecdotes on experimentation that you can share: I also want to curate interesting jokes and anecdotes that can be quickly shared between fellow experimenters. I strongly believe that to build a culture of experimentation, experimenters need to influence the environment with memes that can replicate quickly in the minds of fellow workers. In this regard, the third key takeaway will be these cultural memes that can help us together build a successful culture of experimentation.

I expect to build a community of like-minded people through this blog. I will always be open to comments, questions, and discussions by all readers.

Let’s not be Hubris

In 1849, Cholera had spread through Europe and the common belief was that Cholera spreads through the air. New studies had started to show that probably the cause of Cholera was not the air but bad water. Under the circumstances, a tenant complained to the landlord that many people in the building have gotten Cholera and the water in the tank regularly stinks, so there might be a correlation. The landlord came down to the apartment, filled a glass of water from the tap, and chugged it down to show that he was certain there was nothing wrong with the water. He died three days later.

Hubris is the quality of having an exaggerated confidence in your beliefs, so much so that you are ready to risk your life on it.

Most of us have sought to build our lives on the beliefs and principles that define our individualities. Let us for once take the diversion to understand which of our beliefs are wrong.

It is hard to accept how deep into our belief systems is data ready to prick through. So, let us for once take the diversion to burst all our conceptions before taking a step towards data.

Finally, when an engineer suggests the idea that, 

“Let us add a few words to the search page headlines”, 

Let us for once take the diversion to tell them, 

“Feel free to test all the ideas that you get. When you have a winner we will celebrate.”