Online Experiments For Computational Social Science

Experiments are the gold standard for establishing causal relationships. While Web-based experiments ("A/B tests") have routinely been used to assess alternative ranking models or user interface designs, they have become increasingly popular for answering important questions in the social sciences. This tutorial teaches attendees how to design, plan, implement, and analyze online experiments. First, we review basic concepts in causal inference and motivate the need for experiments. Then we will discuss basic statistical tools to help plan experiments: exploratory analysis, power calculations, and the use of simulation in R. We then discuss statistical methods to estimate causal quantities of interest and construct appropriate confidence intervals. We then discuss how to design and implement online experiments using PlanOut, an open-source toolkit for advanced online experimentation used at Facebook. We will show how to implement a variety of experiments, including basic A/B tests, within-subjects designs, as well as more sophisticated experiments. We demonstrate how experimental designs from social computing literature can be implemented, and then collaboratively plan and implement an experiment together. We then discuss issues with logging and common errors in the deployment and analysis of experiments.Finally, we will conclude the tutorial with a discussion of strategies and scalable methods for analyzing online experiments, including working with weighted data, and data with single and multi-way dependence. Throughout the tutorial, attendees will be given code examples and participate in the planning, implementation, and analysis of a Web application using Python, PlanOut, and R.