Model-Free Error Assessment for Breadth-First Studies, with Applications to Cell-Perturbation Experiments

Experiments adhering to the same protocol can nonetheless lead to different conclusions, for instance, due to batch effects or lab effects. A statistical test applied to measurements from one experiment may yield a vanishingly small $p$-value, yet applying the same test to measurements from a replicate experiment may yield a large $p$-value. Recent work has highlighted this lack of reproducibility in cell-perturbation experiments. We introduce the Reproducible Sign Rate (RSR), a new reproducibility metric for settings in which each hypothesis test has two alternatives (e.g., upregulation and downregulation of gene expression). The RSR identifies the proportion of discoveries that are expected to reproduce in a future replicate. We provide conditions under which the RSR can be estimated accurately -- even when as few as two experimental replicates are available. We also provide conditions under which high RSR implies a low Type S error rate. We demonstrate the uses of RSR with experiments based on several high-throughput technologies, including L1000, Sci-Plex, and CRISPR.