Forced-choice questionnaires (FCQs) are increasingly favored over traditional Likert-type formats due to their reduced susceptibility to faking and social desirability (SD). Their construction typically involves pairing items from existing single-stimulus banks. This study compares four methods for assembling FCQs: a genetic algorithm (GA), two simulated annealing (SA) strategies (blueprint-based and scale-parameter-optimized), and brute-force (BF) random search. These methods are evaluated via simulation and an empirical example, focusing on trait score recovery. The effects of questionnaire length and SD matching on recovery are also examined. Three item banks varying in the a j -SD j relationship and inclusion of heteropolar blocks were used to assess performance across pairing scenarios. GA consistently produced the most reliable scores, followed by SA with a j optimization. All examined factors significantly affected reliability. GA is recommended for FCQ assembly, especially with short questionnaires, no heteropolar blocks, and high a j -SD j correlation.
To statistically test whether two groups or models differ, classifier accuracy is compared. However, common accuracy estimates like cross-validation have unknown distributions, making them unsuitable for statistical inference. Alternatives like permutation tests or train-test splits are computationally expensive and limited to frequentist tests against chance. Independent Validation (IV) is a more flexible alternative providing a known estimate distribution. This enables both conventional hypothesis testing and Bayesian analysis of classifier performance. Although Python is most widely used for machine learning, a Python implementation of IV has been lacking so far. This article introduces such an implementation; beyond the core IV algorithm, the package allows to: (1) plot accuracy against training set size, (2) estimate the posterior distribution of the asymptotic accuracy, and (3) query the posterior for statistics and credible intervals. This makes it easy to apply IV when comparing accuracy posteriors across classes, datasets, or classifiers on the same data.
Scalar invariance is widely regarded as essential for comparing test-score means across groups. However, it is less clear when test scores can be meaningfully compared at the individual level — specifically, whether individuals from different groups who share the same observed score have the same expected value of the latent trait. I show that scalar invariance alone is insufficient for meaningful person-level comparisons based on sum scores. In addition to scalar invariance, person comparison invariance requires equality of latent variable means and omega coefficients across groups. Nevertheless, non-invariance effects can be relatively small if the omega coefficients are high and similar in magnitude across groups. I relate person comparison invariance to predictive invariance and provide R code to test person comparison invariance and to visualise the effects of non-invariance.
Anticlustering has been used as a novel method to assign subjects to conditions in experiments. Anticlustering can be applied when covariate measurements are available at the beginning of an experiment and minimizes differences in covariates between conditions. In a simulation study implementing a two-group between-subjects design, we compared anticlustering with established methods for minimizing covariate imbalance: rerandomization and minimization. Anticlustering most strongly reduced covariate imbalance, followed by rerandomization and minimization. Lower covariate imbalance increased the precision of the effect size estimate. The average statistical power of the unadjusted analysis (independent t-test) was not improved when using covariate-based assignment as compared to random assignment. However, with random assignment, the statistical power of the unadjusted analysis depended on observed covariate imbalance; with covariate-based assignment, the statistical power of the unadjusted analysis was less affected by covariate imbalance because imbalance was minimized. Statistical adjustment via regression was most important to maximize statistical power.
This article introduces the sequential response model with covariates (SRM-C) for analyzing process data, with emphasis on three key capabilities: detecting potential measurement bias in response processes, evaluating group differences in ability distributions and improving parameter estimation precision. The SRM-C combines measurement and structural components, with the measurement component modeling response sequences conditional on abilities and covariates, and the structural component characterizing group-specific ability distributions. Sparsity assumptions implemented through horseshoe prior distributions address identification issues within the Bayesian framework. Monte Carlo simulations demonstrated robust parameter recovery and effective differential item functioning (DIF) detection. An empirical analysis of PISA problem-solving data illustrated the model's utility in distinguishing ability differences from potential measurement bias. The SRM-C offers a comprehensive framework for understanding group differences in process data while ensuring measurement fairness.
Fixed-effects regression models are commonly used in longitudinal studies as a means to estimate causal effects while controlling for unobserved time-invariant confounders. However, unobserved time-varying confounding remains potentially problematic, and identifying and measuring such confounders can be resource-intensive and costly. We propose the Time-Varying Confounding Structural Equation Model (TVC-SEM), a simple longitudinal model that builds on previous "common factor" models and which can serve as a robustness check for the assumption of no unobserved time-varying confounding in the fixed-effects approach. We posit a model with a latent autoregressive variable Zit, which represents the combined influence of both time-invariant and time-varying unobservables, and which is linked to the independent and dependent variables over time. Through Monte Carlo simulations and analyses of data from the Early Childhood Longitudinal Studies Kindergarten cohort (ECLS-K) and the Rural Substance Abuse and Violence Project (RSVP), we show that, under most conditions, TVC-SEM provides less biased estimates than several variants of the traditional fixed-effects model. Our proposed approach offers applied researchers a practical check for gauging the extent to which the fixed-effects assumption of no time-varying confounding may produce bias in the estimation of causal effects.
In time-to-event analyses in social sciences, there often exist endogenous time-varying variables, where the event status is correlated with the trajectory of the covariate itself. Ignoring this endogeneity will result in biased estimates. In the field of biostatistics this issue is tackled by estimating a joint model for longitudinal and time-to-event data as it handles endogenous covariates properly. This method is underused in the social sciences even though it is very useful to model longitudinal and time-to-event processes appropriately. Therefore, this paper provides a gentle introduction to the method of joint models and highlights its advantages for social science research questions. We demonstrate its usage on an example on marital satisfaction and marriage dissolution and compare the results with classical approaches such as a time-to-event model with a time-varying covariate. In addition to demonstrating the method, our results contribute to the understanding of the relationship between marriage satisfaction, marriage dissolution and other covariates.
Single-case designs (SCEDs) assess intervention effects through repeated measurements on one or a few individuals. Multilevel models nest repeated measures within individuals and have gained popularity for inferential analysis in SCEDs, in combination with expert knowledge of the clinicians and applied researchers. However, researchers often face model specification challenges without knowing the true population model underlying their data. This study evaluates how model selection criteria (AIC, BIC, WAIC, LOO) conditioned on the selected model impact statistical power and Type I error rates in intervention effects, reflecting the ecological reality where practitioners do not know the true model. A Monte Carlo simulation modelled data of AB designs varying sample size, measurement points, intervention effects, and random effect structures. Competing multilevel models were then fitted and compared using AIC, BIC, WAIC, and LOO to examine the impact of model selection on statistical power and Type I error rates. Results indicated that frequentist criteria performed well in simpler models in terms of power, while Bayesian approaches showed greater robustness with respect to Type I error control. The findings provide practical insights on multilevel model selection under real-world conditions, highlighting Bayesian methods as a robust alternative for applied researchers handling small sample sizes and complex data structures.
Accurate estimation of standard errors (SEs) is essential in SEM as they quantify the uncertainty of parameter estimates, are fundamental to computing test statistics, and ensure robust population inferences. This study evaluated SEs within the Local Structural-After-Measurement (LSAM) framework, a two-step approach to SEM. Two simulation studies examined analytic and resampling-based SE methods under varying conditions, including normal and nonnormal data, different sample sizes, and both correct and misspecified models. The nonparametric bootstrap yielded near-unbiased SEs under nonnormality, even when models were misspecified, while the parametric bootstrap performed well under normal conditions with correct model specification. The analytic two-step method performed well under normal conditions but showed increased bias with nonnormal data and smaller samples. The robust two-step method reduced this bias in larger samples, though some underestimation remained in small-sample and misspecified conditions. To complement SE bias results, 90% coverage rates were assessed. Findings confirm LSAM’s capability for accurate SE estimation in challenging research contexts.
Understanding how humans observe and interpret actions is vital for social interaction. Point-light displays (PLDs), which depict actions using only joint movements, are widely used to study this process. Recently, PLAViMoP — an open-access database of 3D PLDs covering everyday actions, fine-motor skills, sports movements, facial expressions, social interactions, and robotic actions — has been introduced to facilitate the use of PLDs. PLAViMoP includes a search engine and metadata for each sequence, including movement type, label, actor sex, and age. In complement to the database, here we present a novel methodology that integrates eye-tracking data into the PLD reference frame, allowing gaze behavior and action kinematics to be jointly analyzed (i.e., in a unified dataset). This combined approach offers new insights into action perception and has broad applications in health, sports, and occupational settings. It also offers a promising tool for continuous psychophysical studies of the perception of biological movement.
In behavioral sciences, researchers frequently employ mediation analysis with longitudinal data. A common scenario involves the 1 → 1 → 1 mediation model, where the Predictor X, Mediator(s) M, and Outcome Y are all measured at different occasions (Level 1) within individuals (Level 2). The standard 1 → 1 → 1 mediation model approach fits two multilevel models, one for the mediator and one for the outcome, with two random intercepts and three random slopes in total. However researchers often exclude random slopes from multilevel models and only include random intercepts to account for non-independence across observations of the same individual. We demonstrate that ignoring random slopes in the 1 → 1 → 1 mediation model can result in biased average indirect effect estimators, as well as underestimated standard errors. We provide code from open source and free statistical software that can be used by practitioners to fit the 1 → 1 → 1 mediation model.
Likert scales are widely used to measure agreement levels, typically on a structurally and linguistically symmetrical scale. In non-English settings, literal translations of scale anchors often produce awkward or asymmetrical phrases that deviate from everyday language, potentially affecting data validity. This study examines the impact of translations through an online experiment with 532 Slovenian smartphone users, randomly assigned to two groups. One group used structurally symmetrical translations (GroupSA), while the other used more natural but asymmetrical translations (GroupCA) to measure constructs related to information security within Protection Motivation Theory. GroupCA showed significant correlations with the dependent variable for all predictors, consistently higher Composite Reliability and Average Variance Extracted, slightly higher means, and lower skewness and kurtosis. Additionally, "Completely agree" was chosen more often than the less familiar "Strongly agree." These results highlight the influence of translation choices on survey outcomes, emphasizing the need for careful linguistic adaptation in cross-cultural research.
The aim of the article is to refine a calibration method already presented and used to improve the information provided by the scales of frequency in questionnaires by combining it with data from time use diaries. In other words, this study proposes improvements to an existing calibration method aiming at "adjusting" the data gathered through items - which is useful for the analysis of phenomena with relatively long time cycles, but also notoriously subject to bias - with the data gathered through daily diaries - which are less subject to distortion, but generally only suitable for analysing phenomena with short or very short time cycles. In some cases, in fact, the calibration model already proposed may be problematic since, as we shall see, it could introduce another possible cause of bias. Such distortion could become relevant in certain situations and can be remedied by the proposed refinement with the new calibration model under consideration here. Finally, to highlight the advantages of the proposed method, we will develop with practical applications, comparisons by applying the presented models on data on religious practice collected in a large survey conducted in Italy in 2008. It should be specified, however, that the applicability of the proposed model is not limited to this example and can be extended to other contexts and types of data.
This paper presents the SUSHIJA (Scoping, Updated, Systematic, Holistic, Interpretive, Joint, and Adaptive) Framework for conducting scoping reviews in an innovative manner. The framework is developed to address limitations of traditional scoping reviews. The SUSHIJA framework has several inclusive features such as Artificial Intelligence (AI) driven automation of literature reviews and data extraction, critical appraisal of the involved literature, iterative thematic mapping of the included articles, and even a living review component to extract and synthesize new studies in real-time. SUSHIJA framework also considers stakeholder input and incorporates visual tools.
Scientific claims, and the way they are tested, must be unambiguous and flexibility must be disclosed. Grounded in Popper's principle of falsification, we suggest the Falsification Assessment Form (FAF). The form aims to identify ambiguity and undisclosed flexibility in the entire research process with 11 items covering hypothesis formulation, data processing, analysis, and alternative explanations. It also collects information on transparency measures, such as preregistration. The form was developed through consensus among the authors and refined via a collaborative feedback assessment of 19 experts. It is intended for original, quantitative research, it highlights potential issues and requires authors to provide detailed responses. FAF is meant to be a structured qualitative audit framework. It can be used to identify concerns in published research, improve the quality of papers during peer review, or guide rigorous study planning from the outset. We open up further refinement and testing of FAF to the scientific community.
Dose-response relationships between continuous risk factors and disease outcomes are necessary for understanding the risks related to different levels of exposure. Dose-response risk curves can lead to more targeted public health messaging, prevention efforts, and policy implementation. Meta-analyses are often used to combine statistical results from different studies and can be used to model dose-response relationships. However, several challenges are encountered when performing dose-response meta-analysis, such as having heterogeneous reference categories, inconsistent measures of risk, and determining the most accurate shape of the curve. In this paper, we propose a three-step process for estimating dose-response relationships via meta-analysis, which involves: 1) harmonizing the measures of risk, 2) homogenizing the reference category, and 3) selecting meta-regression models. We use data obtained from a systematic review on the dose-response relationship between alcohol consumption and the risk of chronic liver disease to provide an example of the proposed process.
Specifying complex time series models typically allows for a wide range of plausible analysis strategies. However, researchers typically perform and report only a single, preferred analysis while ignoring alternatives that could yield different conclusions. As a remedy, we propose multiverse analysis to investigate the robustness of dynamic network analysis to arbitrary modeling choices. We focus on group iterative multiple model estimation (GIMME), a highly data-driven approach, and re-analyze two datasets (combined n = 199). We vary seven modeling parameters, resulting in 3,888 fitted models. Group-level and to a lesser extent subgroup-level results were mostly stable. Individual-level estimates were more heterogeneous, with some decisions strongly influencing results and conclusions. The robustness of GIMME to alternative modeling choices depends on the level of analysis. For some individuals, results may differ strongly even when changing the algorithm only slightly. Multiverse analysis is a valuable tool for checking the robustness of results from time series models.
When analyzing treatment effects on outcome variables constructed from psychometric instruments (e.g., educational test scores, psychological surveys, or patient reported outcomes), researchers face many choices and competing guidance for scoring the measures and modeling results. This study examines the impact of outcome measure scoring and modeling approaches through simulation and an empirical application. Results show that estimates from multiple methods applied to the same data will vary because two-step models using sum or factor scores provide attenuated standardized treatment effects compared to latent variable models. This bias dominates any other differences between models or features of the data generating process, such as the use of scoring weights. An errors-in-variables (EIV) correction removes the bias from two-step models. An empirical application to 10 datasets from randomized controlled trials demonstrates the sensitivity of the results to model selection. This study shows that the psychometric principles most consequential in causal inference are related to attenuation bias rather than optimal outcome scoring weights.
This article introduces the Appreciative Qualitative Interview, a qualitative research technique grounded in Appreciative Inquiry, Social Constructionism, and the narrative turn in psychology. The Appreciative Qualitative Interview emphasizes positive, strength-based questioning to elicit more empowering personal and collective narratives from participants in qualitative research studies. The aim is to uncover and amplify participants' successes, strengths, and aspirations, thereby facilitating a more generative, enabling, and transformative research process. This tutorial article provides a detailed interviewing blueprint for conducting Appreciative Qualitative Interviews, including opening, core, and closing questions. While acknowledging the challenges of overemphasizing positivity, the article highlights the value of this approach within an action research and social constructionist epistemology.
Principal component analysis is a well known method for dimension reduction based on the covariance matrix associated to a multivariate data table. Therefore, a large amount of work has been devoted to analyzing the sensitivity of the eigenstructure of this matrix to influential observations. In order to evaluate the effect of deleting one or a small subset of observations, several approximations for the perturbed eigenelements have been proposed. This paper provides a theoretical and numerical comparison of the main approximations. A special emphasis is given to those based on Rayleigh quotients since they are under-utilized given their excellent performance. A general approach, using refined inequalities, is proposed in order to get a precise evaluation of their accuracy without having to recompute the exact perturbed eigenvalues and eigenvectors. This approach is of specific interest from a computational standpoint. Theoretical developments are illustrated with a numerical study which emphasizes the accuracy of approximations based on Rayleigh quotients.
