You Can't Always Get What You Want: Using 'Broken Lotteries' to Check the Validity of Charter School Evaluations Using Matching Designs

We consider situations in which public charter school lotteries are neither universally conducted nor consistently documented. Such lotteries produce “broken” Randomized Control Trials, but provide opportunities to assess the internal validity of quasi-experimental research designs. Here, we present the results of a statewide charter school evaluation using a broad-based student matching evaluation design, and run two additional analyses using the charter application waitlists as robustness checks. Our additional models, which address concerns of self-selection by using only charter applicants as matched comparison students, yield similar effect estimates and thus provide support for the use of matching designs in charter school evaluations.