Online controlled experiments: introduction, insights, scaling, and humbling statistics.

ABSTRACTThe web provides an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using controlled experiments (e.g., A/B tests and their generalizations). From front-end user-interface changes to backend algorithms, online controlled experiments are now utilized to make data-driven decisions at many other companies. While the theory of a controlled experiment is simple, running online controlled experiments at scale - hundreds of concurrent experiments on a given day at Bing has taught us many lessons. We provide an introduction, share real examples, key insights, cultural challenges, scaling challenges, and humbling statistics.