Network analysis has become a popular computational-modeling approach in psychological research and is now increasingly used to address confirmatory research questions. However, preregistration of such analyses remains rare and even when implemented may leave room for researcher degrees of freedom, particularly when data-dependent decisions are difficult to fully anticipate. In this tutorial, we demonstrate how researchers can safeguard against bias by using analysis blinding as a complementary step to preregistration. The method involves temporarily altering the data to remove the key effect of interest while preserving all other aspects. Analysis blinding makes it possible to explore important features of the data, for instance, spotting outliers or adjusting the computational model, without introducing bias. In this tutorial, we present three common research questions in network analysis and show how they can be addressed using analysis blinding to safeguard their confirmatory nature. We use data from previously published studies employing network analysis and offer a practical, step-by-step guide along with R code that researchers can adapt to their own data and research questions.
A pilot study is a preliminary investigation conducted before a full-scale research study. Although piloting is a common and valuable part of the research process, pilot studies are not systematically reported (or even acknowledged) in the psychology literature. In this article, we argue that establishing norms for regularly acknowledging preliminary research phases can improve many aspects of psychological research. Specifically, reporting pilot-study information can help researchers provide a more complete record of the research process, reveal selection biases in research designs that may affect the generalizability of findings, evaluate others’ research more accurately, learn from other researchers’ successes and failures, uphold ethical responsibilities, and improve the methodological practice of piloting. We conclude with three concrete suggestions for near-term changes to help pilot reporting become standard practice: Scientific communities and institutions should develop simple pilot-reporting templates, journals should provide clear pilot-reporting guidance for authors, and authors should acknowledge when pilot studies were run and share relevant materials and data. Current reporting standards favor conclusive and polished research narratives, but such narratives are the end product of a complex scientific process that should also be shared. A truly transparent science should spotlight processes, such as piloting, that underpin the conclusions scientific articles draw.
“Stereotype threat” refers to the fear of being judged based on negative stereotypes about the performance of a certain group one identifies with. Numerous published studies have found that stereotype threat might lower mathematics-test performance among women. However, many studies have used suboptimal designs and analyses, the literature might be subject to publication bias, and the cross-cultural generalizability of the effect remains unknown. In this registered replication report, we describe the result of nine direct replications (total N = 1,709) of a representative stereotype-threat study by Johns et al., who found that threatened women (but not men) underperform when they are confronted with a mathematics test that is presented to measure gender differences and that this effect can be alleviated by altering test instructions. A multilevel analysis showed no significant interaction effect between stereotype-threat condition and gender and an observed gender gap only on the mathematics test, in which women performed worse than men (Hedges’s g = −0.26, SE = 0.11, 95% confidence interval [CI] = [−0.47, −0.05]). The stereotype-threat effect among women was virtually null and not significant ( N = 1,275, g = 0.04, SE = 0.08, 95% CI = [−0.11, 0.20]) and considerably smaller than the original study ( N = 45, d = −0.82, 95% CI = [−1.45, −0.18]). Secondary analyses yielded a main effect of gender composition and a random effect of lab on mathematics-test performance, but neither variable affected the interaction between stereotype threat and gender. The current results fail to replicate the stereotype-threat effect by Johns et al., hence casting doubt on the generalizability of the effect of stereotype threat on women’s mathematics performance.
Sample-size determination (SSD) is the procedure of determining the number of subjects necessary to achieve a desired level of statistical power and is essential in planning an experiment. Although open-access software for SSD via closed-form equations is readily available in the null-hypothesis-significance-testing framework, this approach has been subject to severe criticism in the past. As an alternative, Bayesian evaluation of informative hypotheses via the Bayes’s factor or posterior model probabilities has been proposed. However, available software packages for Bayesian SSD are either (a) limited to simpler models, such as analysis of variance and t test, and cannot handle longitudinal data or (b) unable to handle more than two treatment conditions. Current software also neglects participant attrition—a common occurrence that may substantially reduce the power of a longitudinal experiment. In the present work, we address this gap by introducing the open-access R package BayesSSD , which performs simulation-based Bayesian SSD for longitudinal trials with two or more treatment conditions. Through a simulation study, we show that not only the proportion of individuals dropping out but also the timing of dropout need to be considered when performing SSD. In the presented method, various patterns of expected attrition can be specified via parametric and nonparametric survival functions and accounted for in the SSD procedure. To facilitate adoption, we provide a tutorial with empirical data, illustrating each step of the Bayesian SSD process.
