A growing trend is to launch new products and features through A/B tests where units are randomly assigned to treatment and control. While A/B tests are often referred to as the “gold standard” in measuring causal impacts, poor design can lead to inconclusive results and violate core assumptions. This talk covers the full process of running A/B tests to drive business decisions.
We discuss steps before a test, including getting stakeholder buy-in, outlining actionable launch criteria, and experiment design. We then present action items during the test to ensure successful randomization and finally, we go over how to conduct the analysis and how to turn (potentially mixed) statistics into business recommendations.