The History and Philosophy of Experimentation
If science had to be explained in the fewest words possible, I would choose to call it “the discovery of causality”. The finest of our instincts and intelligence often guide us towards asking why things happen and what would happen if those things had not happened. It seems as if underlying the laws of nature there is a causal graph that links causes to their effects. Finding and understanding these causal links to design systems that harness the power of causality is the primary pursuit of science.
Observations and data are the inputs to our study of causality but data has a sharp way of deceiving us. One usually starts to look at things that happen together in the dataset to get a peek into what is causing what. But what happens when they see prices of tomatoes being correlated to prices of cars (both are caused by inflation), and children’s shoe sizes being correlated to the size of their bags (both are caused by age), and finally, the correlation between the number of birds and the number of babies in the city (both caused by settlements)? As it happens, more often than not two seemingly unrelated things come out to be correlated in data. See this link for interesting spurious correlations.
Observational data is never enough to infer causality. What one needs to infer causality is an experiment. An experiment that lets you control everything else and just change one cause and see its effect. While A/B testing is a 21st century concept, experimentation has been the cornerstone of science for several hundred years. Understanding the history and the philosophy behind experimentation is crucial for building a broader scientific mindset. This blog post introduces the reader to the ideas in the history and philosophy behind experimentation.
History of Experimentation
Causal hypotheses probably started with crude pattern recognition techniques looking at similarities and correlations between different phenomena. For instance, we know that the earliest humans tried to infer causal relationships between the patterns of the stars and natural events. Such hypotheses often get tested in the long haul as people tend to remember patterns of thoughts that work and tend to forget those that do not seem to be true. Such a procedure of science was at best ad hoc, until the formalization of experiments in the last 1000 years.
Scientific inquiry matured with the idea of experimentation as scientists started to prove or disprove their hypothesis by actually creating an environment where everything else could be fixed and just the cause could toggled to see the effect. This was called a controlled environment and in its absence, it was hard to understand if the effect in place actually happened due to the toggled cause or something else that has changed in the environment. But as the inquiries of science spread to different domains like medicine, social science, and psychology, it was realized that creating a controlled environment was a rare luxury in itself. The world is uncertain and humans could go only so far in controlling its stochastic nature.
A major breakthrough in experimentation was made when in the early 20th century, RA Fisher gave birth to the Randomised Controlled Trials (RCT) that still are the gold standard of causal discovery. RCTs solved the problem of natural randomness by randomly assigning subjects to a control group and the treatment group. If the assignment was truly random, Fisher proved that all other environmental factors will get equally divided into the two groups and the only difference that will remain is the treatment (the cause). Modern A/B testing is fundamentally an RCT run on a website.
Experimentation in the 21st century, specifically in the digital domain, has been made easy for everyone. Anyone can today run an A/B test on their website and start studying what causes their customers to convert. In the future, I expect generative AI will take the place of generating ideas, and experimentation will be automated to ensure that only those ideas survive that create the desired effect. Humans will largely become the meta-drivers deciding the direction of evolution by carefully crafting metrics to measure and optimize.
Philosophy of Experimentation
Where does knowledge come from? David Deutsch reflectively asks this question and tries to answer the same in his book, The Beginning of Infinity. For the longest time before the scientific revolution happened, people relied on religious texts for the answers to one of their deepest questions. The propagators of religion held on to power for centuries by providing an interpretation of this text that suited their interests. The scientific revolution put an end to this belief as people started to rely on experience as a more reliable source of truth. Empiricism was born.
Empiricism was the doctrine that truth comes from experience and was valuable because for the first time, it put an end to religious supremacy. Empiricism later evolved into inductivism, where people started to believe that repetition was the key, and more often something happened in the past, the more likely it will be true in the future as well. But experience could only take humans so far as Immanuel Kant went on to argue that experience was mostly personal and often far from the reality of the world. Experience was definitely the source of some knowledge but not all.
We finally learned that the source of knowledge is human creativity and human thought. A human looks at the various pieces of information from experience and then creates a hypothesis of how a phenomenon occurs. He then goes on to design an experiment that can either prove or disprove that hypothesis. This goes on in a cycle as humans go on disproving previous hypotheses and replacing them with new hypotheses supported by experimentation. Karl Popper in the 20th century formalized this theory with the theory of falsification which says that anything that cannot be proved wrong by evidence is not scientific.
Hence, knowledge originates with a hypothesis, and never can a hypothesis be proved to be 100% true.
Conclusion
The curious experimenter might ask how knowing the history and philosophy of experimentation helps me in my work. And yes, this question has been inflicted upon history and philosophy many times in different domains.
History plays its role in giving us a wider context of what we are doing. The wider context becomes relevant when we try to understand why things happen the way they do, and which mistakes to not make again. I feel that experimenters and scientists often find themselves on the front lines of innovation where no clearly cutout path is available to follow. History comes to the rescue here as the one who knows history deeply understands why things came to be the way they are and which would be the best path ahead.
Philosophy, on the other hand, is the guiding light of science. While science is the discovery of causality, it is a haphazard random walk unless you really understand where you are going. Philosophy attempts to answer some of the deepest questions related to why and more importantly makes us aware of some of the deepest questions we never thought of asking. For instance, Deborah O’Malley once asked me such a question that I had never thought of, “Is there a checklist we can go through in an A/B test that will prove with 100% confidence that the test results are true?” The short answer to this question, buried inside experimentational philosophy, is no.
Once you understand the history and philosophy behind experimentation, you naturally build the scientific mindset, imbibe it into daily life and then let it guide your work.
