To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N > 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = -0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

Will people follow their intuition even when they explicitly recognize that it is irrational to do so? Dual-process models of judgment and decision making are often based on the assumption that the correction of errors necessarily follows the detection of errors. But this assumption does not always hold. People can explicitly recognize that their intuitive judgment is wrong but nevertheless maintain it, a phenomenon known as acquiescence. Although anecdotes and experimental studies suggest that acquiescence occurs, the empirical case for acquiescence has not been definitively established. In four studiesusing the ratio-bias paradigm, a lottery exchange game, blackjack, and a football coaching decisionwe tested acquiescence using recently established criteria. We provide clear empirical support for acquiescence: People can have a faulty intuitive belief about the world (Criterion 1), acknowledge the belief is irrational (Criterion 2), but follow their intuition nonetheless (Criterion 3)even at a cost.

The use of performance enhancing drugs (PEDs) elicits widespread normative opposition, yet little research has investigated what underlies these judgments. We examine this question comprehensively, across 13 studies. We first test the hypothesis that opposition to PED use cannot be fully accounted for by considerations of fairness. We then test the influence of 10 other potential drivers of opposition in an exploratory manner. We find that health risks for the user and rules and laws prohibiting use of anabolic steroids reliably affect normative judgments. Next, we test whether these patterns generalize to a different PED-cognitive-enhancement drugs. Finally, we sketch a framework for understanding these results, borrowing from Social Domain Theory (e.g., Turiel, 1983). We argue that PED use exemplifies a class of violations with properties of moral, conventional, and prudential offenses. This research sheds light on a widespread, but understudied, normative judgment, and illustrates the utility of exploratory methods.