All Posts
A Critique of the Frequentist and the Bayesian Approach
A discussion of the criticisms that the two statistical factions have towards each other: Frequentists, who get criticised for p-values, and the Bayesians, who find it difficult to define the priors….
The Bayesian Way To Statistical Significance
A deep dive into the Bayesian solution to the problem of statistical significance. The article explains: the Prior, the Bayes Rule and the Posterior….
Characteristics of Good Metrics
11 questions to ask when designing metrics corresponding to the desirable characteristics a metric should have. The post covers three major aspects: statistical, logistical and usability….
Three Kinds of Metrics: The Success, The Guardrail and The Diagnostic
The three kinds of metric that serve different purposes in experimentation. Understand how modern businesses develop an arsenal of metrics to guide their experimentation efforts….
Metrics in Experimentation
What are metrics and why are they needed in experimentation? An introduction to the subsection on metrics….
The Frequentist Way to Statistical Significance
A deep dive into the Frequentist solution to solve problems of statistical inference. The article explains the elements common to all Frequentist methods: the null hypothesis, the statistic and the p-…
The Fidelity of Experimentation
Fidelity in research refers to how closely the experimental setup mimics the real world. Understand the implications of fidelity in modern day A/B teting….
The Frequentists, The Bayesians and The Epistemologists
VWO’s learnings on the fundamentals of the Bayesian vs Frequentist Debate. The differences between the two schools of thought and why does it matter….
The Forward Probability and The Backward Probability
A perspective on two different kinds of probabilities that are at the crux of the difference between the Bayesian and the Frequentist school of thought….
Falsification, Experimentation, and Popper
What is the dividing line between scientific and non-scientific? An introduction to the theory of falsification and its implications on experimentation….
The Fourth Factor in Statistical Significance
An introduction to base rates – the fundamental difference between Bayesian and Frequentist schools of thought. Understand when and why base rates matter in statistical testing….
The Arbitrary, The Random and The Pseudorandom
An account of the nuances of randomness in modern computing and how it matters in A/B testing….
The Relationship Behind Statistical Significance
The statistical explanation behind how the different forces of statistical significance are related to each other….
The Forces Behind Statistical Significance
An intuitive explanation of the three forces behind statistical significance: effect size, standard deviation and sample size….
Limitations of Randomized Controlled Trials
A discussion on the fundamental limitations of Randomised Controlled Trials and how do they apply to modern day A/B testing….
The Statistical Significance
An introduction to statistical significance, what does it mean and why do we need it in experimentation….
The Birth of Randomized Controlled Trials
An introduction to where do modern A/B tests come from and what problem did they solve in the larger scientific context….
The Statistics of Experimentation
The larger framework of assumptions and uncertainties that statistics has been built upon and how does a statistician navigate through these problems….
The History and Philosophy of Experimentation
An introduction to the historical context of experimentation and the deeper philosophical questions it has evolved through….
Topics to Discuss in Experimentation
A detailed summary of the structure of this blog, the broader subsections it will have and the ideas I will discuss….
Experimentation, Data Science, and Statistics at VWO
The research culture in the Data Science team at VWO and the story of how we learned all that we plan to share in this blog….
Introduction: Why read this blog?
Introduction to the three pillars of this blog and the three key takeaways for the modern experimenter….