On the statistical analysis of experiments with manipulation checks

Social scientists are often interested in the effect of an internal state, such as ego depletion or social stress, that cannot be directly assigned in an experiment. Instead, they might assign participants to a manipulation intended to produce this internal state, using manipulation checks to assess whether the manipulation did so effectively. We discuss statistical analyses for such experiments, focusing on cases where researchers are primarily interested in the average treatment effect (ATE) of the target internal state rather than that of the manipulation. Often, researchers estimate the association of the manipulation itself with the dependent variable, but this intention-to-treat estimator is typically biased for the ATE of the target state, and the bias could be either toward the null (conservative) or away from the null. We discuss the assumptions under which this estimator is conservative. In fact, under a somewhat weaker version of the same assumptions, one can alternatively use a consistent estimator for the effect of the target state. We discuss how to do so using standard instrumental variables analysis. Conducting such an analysis along with the standard intention-to-treat analysis would more directly address the central question of interest when using manipulation checks. As a running example, we consider a multi-site replication study on the ego-depletion effect, in which the manipulation’s partial effectiveness led to criticism and several re-analyses that arrived at varying conclusions. We use instrumental variables analysis to directly account for the manipulation’s partial effectiveness; this corroborated the replication authors’ reported null results.