A Tutorial on Safe Anytime-Valid Inference: Practical Maximally Flexible Sampling Designs for Experiments Based on E-Values

We show how e-values simplify the design and the conduct of experiments. These e-values yield anytime-valid tests and confidence intervals that preserve type I error guarantee regardless of the sample size. This enables real-time monitoring of evidence as data are collected, permitting early termination of experiments without intolerably inflating the risk of making a false discovery. Early stopping not only preserves resources, but also mitigates risk for participants in clinical settings. Anytime-valid tests always allow for optional continuation, that is, the extension of an experiment regardless of the motivation. For instance, if more funds become available, or if the evidence looks promising and the funding agency, a reviewer, or an editor urges the experimenter to collect more data. Analogously, a researcher can be assured that a 95% anytime-valid confidence interval will, with at least 95% chance, cover the true effect size regardless of how, or even if, data collection is stopped. We use the free and open-source software library safestats implemented in R to illustrate the practical benefits of this novel inference framework.