The I-squared statistic has become hugely popular in meta-analysis. Motivated by the recent statement that 'Informally, it can be considered to reflect the extent to which confidence intervals displayed in a forest plot do not overlap with each other', this short note formalises this perspective. The insights provided by our analysis lend further weight to the interpretation of the I-squared statistic as a measure of inconsistency. This perspective reminds us that visual inspections of forest plots are vitally important when assessing between-study variation.
Abstract Meta-analysis is often viewed as a gold standard for synthesizing available evidence, but it faces challenges as evidence becomes available gradually and studies may be disseminated selectively and at different times. When publication timing is systematically related to study results, meta-analytic conclusions at any given time can be biased; for example, studies with statistically significant or larger effects may be published earlier than those with null or smaller effects. This phenomenon, commonly referred to as time-lag bias, is a key reason why early meta-analytic conclusions may later weaken, reverse, or become more heterogeneous as additional evidence accrues. To address the limitations of existing definitions and detection approaches, which are largely tied to monotone, effect-size-driven dissemination, we broaden the concept of time-lag bias beyond the traditional yet simple notion that “positive” studies are published earlier. Specifically, we view time-lag bias more generally as a manifestation of time-dependent evidence availability, whereby the timing of study dissemination can distort meta-analytic conclusions at different stages of evidence accrual. We illustrate this broader perspective through several real-world case studies, including classical time-lag bias, inverse time-lag bias, non-monotone temporal patterns, and anomalous evidence localized within specific time windows. These examples show that time-lag bias can arise through multiple dissemination mechanisms and may exaggerate, attenuate, or even reverse pooled estimates over time. They highlight the importance of explicitly accounting for the temporal dimension of evidence accumulation and motivate the routine use of diagnostic and sensitivity strategies to assess robustness to selective dissemination over time.
Abstract Meta-analytic structural equation modeling (MASEM) is a flexible modeling framework that combines the strengths of meta-analysis and structural equation modeling. Despite the increasing popularity of the MASEM technique, its usage and scope are limited to relatively simple meta-analytic data structures and moderator hypotheses. Within-study variability in moderator variables and dependence in primary studies’ correlation estimates tend to be handled using ad hoc methods. In this study, we demonstrate use of multilevel multivariate modeling with robust variance estimation (RVE) for handling within-study dependence when pooling studies’ correlation estimates and we propose how to incorporate categorical moderator values directly into the structural equation model for moderators with values varying within studies. We outline the proposed approach using three stages: (a) specification of the structural equation model using moderator variable categories, (b) pooling correlation matrix elements using the three-level multivariate model with RVE, and (c) testing the moderator hypotheses through either model comparison or direct estimation of structural equation model’s path differences. We illustrate the approach using three examples to demonstrate its flexibility across different MASEM models. This framework broadens the scope of moderator hypotheses that can be tested using MASEM and enhances the methodological rigor for handling dependencies in studies’ correlation estimates for MASEM.
Abstract Evidence synthesis (ES) is costly and time-intensive. Artificial intelligence (AI) tools are increasingly used to automate or semi-automate ES tasks. We conducted a living rapid review of AI tools that automate or semi-automate ES tasks. We searched for published articles and preprints through MEDLINE, Embase, and the Association for Computing Machinery Digital Library from January 1, 2021 through October 15, 2025, supplemented by hand-searching. We included existing reviews to capture evaluations before 2021. Screening was conducted in duplicate at the title/abstract level, with single full-text screening and data extraction verified by a second reviewer. We included 134 studies. Performance varied by task and evaluation approach. Search automation tools showed poor recall (median 17%), although human-in-the-loop approaches improved performance. Tools for identifying randomized controlled trials demonstrated high median recall (98%) and precision (92%). Semi-automated abstract screening tools using active learning reduced screening burden by approximately 50% while maintaining 95% recall. Large language models demonstrated variable accuracy for abstract screening (median 88% recall), full-text screening (median 96% recall), and data extraction (median 63% accuracy). Risk of bias (RoB) assessment tools showed moderate agreement with human reviewers for the Cochrane RoB tool version 1 (median 71%) but performed less well with other RoB tools. AI tools can meaningfully reduce burden in study design identification and abstract screening but perform inconsistently across other evidence synthesis tasks. Tools for search automation, data extraction, and RoB assessment require further development. No existing tool can currently replace human expertise.
Automation-assisted title-abstract screening is routinely evaluated using metrics such as recall, precision, and workload reduction, and results are increasingly summarized across studies. Synthesis presupposes that evaluation results refer to comparable target quantities. This article examines whether screening evaluation results produced under diverse automation-assisted workflows can meaningfully support cumulative inference, and under what conditions cross-study comparison is warranted. This conceptual article treats screening evaluation targets as workflow-induced rather than task-inherent. Evaluation design is decomposed into four workflow dimensions (representation, inference, governance, and evaluation) plus the reference-decision set produced or specified. Recurring choice patterns across these elements define evaluation configurations. Two study-level configurations and one synthesis-level pattern are illustrated: endogenous adjudication, inherited adjudication, and aggregation of heterogeneous targets. Across configurations, evaluation targets are induced by workflow-specific reference-decision sets and, in some designs, model-based target domains rather than by a common reference set. Nominally similar metrics can therefore estimate different quantities across studies, and apparent cumulativeness can arise from pooling estimates of different quantities. Standard heterogeneity statistics cannot distinguish variation in method effectiveness from variation in what is being estimated. Comparison is warranted only when targets align before pooling. Differences across screening evaluations may reflect differences in what is being estimated rather than statistical heterogeneity around a common estimand. When evaluation targets are not aligned across the framework, aggregate summaries can describe reported results but cannot support cumulative inference. Improved aggregation requires both better study-level specification of evaluation targets and explicit attention to configuration variation at synthesis.
