Replication does not reliably measure scientific productivity

Replication surveys are becoming a common tool for assessing the knowledge production of scientific disciplines. In psychology, economics, and preclinical cancer biology, replication rates near 50% have been argued as evidence the disciplines are not reliably producing knowledge, are rife with questionable research practices, and warrant reform. Concerns over failed replications have eroded faith in science, with claims that the vast majority of published research is false. However, these claims are often made under the assumption that effect sizes are fixed and point null hypotheses can be true in practice. Here we derive a theoretical model of the publication process that instead accounts for variation in observed effect sizes. We show that replication rates provide little insight into whether a scientific discipline is reliably and efficiently producing knowledge. In applying our model to data from a large-scale replication survey, we reveal that concerns over the reliability of scientific research may be overstated. Finally, we highlight how proposed reforms may be ineffective at improving replicability and can be detrimental to orthogonal measures of scientific productivity.