Abstract Objectives Systematic evaluation of variability in large language model (LLM)-generated outputs is critical for assessing their reliability. We developed a general statistical framework and Python package (llmReproScore) for quantifying the repeatability and reproducibility of LLM outputs. Materials and Methods Following US Food and Drug Administration terminology, our framework establishes four metrics for quantifying LLM repeatability and reproducibility along two dimensions: semantic and internal. Semantic metrics quantify variability in the meaning of LLM-generated outputs, whereas internal metrics quantify variability in token-generation certainty. We validated these metrics, and as an illustrative use case, applied them to diagnostic reasoning outputs generated across LLMs and prompting strategies using US Medical Licensing Examination questions and 90 real-world rare disease cases from the Undiagnosed Diseases Network. Results Validation results showed that semantic scores aligned with human ratings (Spearman ρ = 0.88 for repeatability and 0.66 for reproducibility) and confirmed that internal metrics capture variability in token-generation certainty. In the illustrative use case, LLM repeatability was generally not associated with accuracy, suggesting that LLM output variability and accuracy capture distinct dimensions of model performance. Discussion The proposed metrics can be used to characterize LLM output variability and systematically compare models, prompts, and generation settings. These comparisons can inform more rigorous selection of LLM configurations based on repeatability and reproducibility. Conclusion We developed and validated generalizable metrics for quantifying LLM output variability. The proposed metrics can complement accuracy-based evaluation to inform experimental design and support more rigorous comparison and selection of models, prompts, and generation settings.
Abstract Background The clinical practice guidelines (CPGs) published by the European Society for Vascular Surgery (ESVS) span approximately 1100 pages whose simultaneous cross-referencing is impractical at the point of care. Unguarded large language models (LLMs) carry hallucination risk, motivating a guideline-locked design. Methods We describe the architecture and development of a context-aware clinical agent encompassing the full ESVS corpus of 14 CPGs. The system extends a validated single-guideline GraphRAG prototype with an LLM-driven Guideline Router, an embedded Context Gate that withholds retrieval when decision-critical parameters are absent, dual-channel retrieval, and GPT-5 synthesis under a mandatory citation template. An automated benchmark assessed the final system on 55 vignettes, scored by 2 blinded LLM judges over 3 runs (330 judgments), with clinician adjudication of every flagged claim. Results The system stayed within its locked corpus, declined all out-of-scope queries, and achieved 98.4% guideline-routing recall. The Context Gate showed 91.9% specificity and 78.6% sensitivity for underspecified queries, and all 259 cited recommendations were verifiable within the locked corpus. Automated judges flagged an unsupported claim in 18.2% of items under a sensitive screening definition; on adjudication by 2 vascular surgeons, 7.3% of items contained a clinically consequential error, none graded capable of causing harm. Clinical acceptability of complete answers was not scored and is reserved for prospective expert evaluation. Conclusion These results support the feasibility of a multi-guideline context-aware clinical agent for vascular surgery; citation traceability and locked-corpus provenance facilitate future auditability.
Abstract Objective To compare patient trust in educational health contentand the physician providing it, when authorship is attributed to a physician or generative artificial intelligence (AI). Materials and Methods In a preregistered randomized, parallel survey study, participants read a health education article. Individuals were randomized to 1 of 4 conditions, attributing authorship of an identical article to (1) a physician author, (2) generative AI, (3) a physician assisted by AI (hybrid authorship), or (4) no author. Pregnant or postpartum English-speaking US adults (n = 400) were recruited in August-October 2025 via an online survey panel. Eighty percent passed timing and attention checks. Participants rated article engagement, content trust, and trust in the (hypothetical) recommending physician with Likert-scales. Bonferroni corrections were applied to post hoc pairwise comparisons (α = 0.05). Results Among participants who correctly recalled authorship (n = 195), AI-only authorship was associated with lower trust in content (mean, 9.43 vs 11.11; P = .001) and the recommending physician (mean, 8.93 vs 10.84; P = .03) compared with physician-only authorship, while hybrid authorship did not differ significantly from physician-only authorship for either outcome (content: P = .08; physician: P > .99). Sensitivity analyses including all participants (n = 321) were generally consistent with the primary findings. Engagement did not differ significantly across conditions in the primary analytic sample. Discussion As generative AI becomes increasingly integrated into clinical care, how AI involvement and physician oversight are communicated to patients could have important implications for patient trust. Conclusion AI authorship was associated with reduced trust in patient-facing education despite identical, clinician-written content.
Objectives To externally validate a pediatric surgical site infection (SSI) prediction model, assess institutional and temporal transportability, and evaluate recalibration strategies to support adaptation of registry-derived predictions for clinical decision support (CDS).Materials and Methods An elastic-net logistic regression model developed using a national pediatric surgical registry (2012-2022) was validated in an institutional electronic health record (EHR)-derived cohort (10 450 procedures) and in a newly released, temporally distinct cohort from the same registry (301 999 procedures). We assessed discrimination, calibration, and overall performance, used a closed testing procedure to guide recalibration, and evaluated clinical utility using decision curve analysis.Results Performance was preserved in the registry cohort, with c-statistic 0.75, Brier score 0.03, calibration slope 1.00, and observed/expected (O/E) ratio 1.01. Direct application to the institutional EHR cohort produced inflated predicted risk and poor performance, with c-statistic 0.60, Brier score 0.07, calibration slope 0.15, and O/E ratio 0.25. After omitting open-ended diagnosis-code indicators, performance improved, with c-statistic 0.70, Brier score 0.03, calibration slope 0.81, and O/E ratio 1.35. Logistic recalibration restored calibration slope and O/E ratio to 1.00. Decision curve analysis showed positive net benefit across prespecified thresholds.Discussion These findings highlight a key challenge for clinical translation: models that perform well under internal or registry-based validation may not be directly transportable to real-world EHR environments. Predictor misalignment-differences in how model predictors were captured and measured across registry and EHR data-emerged as a key barrier to direct institutional deployment, underscoring the importance of implementation-aware external validation before clinical use.Conclusion Addressing diagnosis-code predictor misalignment and applying parsimonious recalibration supported institutional adaptation of a pediatric SSI prediction model, providing a pragmatic pathway for translating registry-derived models into real-world clinical practice. This work sets the foundation for prospective evaluation of EHR-embedded CDS to enable individualized SSI risk estimation at the point of care.
Objective Pregnant and breastfeeding women, along with children, represent populations persistently underrepresented in clinical research, leading to uncertainty in clinical decision-making. This study investigated the landscape of pharmacology research concerning pharmacokinetics (PK), pharmaco-epidemiology (PE), and clinical trials (CT) within maternal and pediatric patient populations.Materials and Methods A pharmacology research landscape in maternal and pediatric patient populations was analyzed using a novel co-learning framework applied to 38 million PubMed abstracts. It incorporated an evolving annotation guideline for abstract labeling, uncertain sampling, annotation error calibration, and large language model training. The co-learning framework enabled human annotators and language models to iteratively learn from each other, optimizing the classification of PK/PE/CT papers. Analysis was restricted to medications prescribed for maternal and pediatric patients, focusing on publication evidence.Results Co-learning significantly increased the F1-scores for classifying the CT, PE, and PK abstracts from (0.81, 0.84, 0.73) to (0.96, 0.94, 0.93), respectively. On the external validation set, the F1-score increased from (0.81, 0.60, 0.42) to (0.95, 0.88, 0.91) for CT, PE, and PK, respectively. The models identified 376 231 and 879 936 abstracts relevant to maternal and pediatric patients, respectively. In pregnant or postpartum women, 22%-66% prescribed drugs lacked either PK or PE/CT evidence. Among pediatric patients, 8%-40% prescribed drugs had no or weak PK or PE/CT evidence.Discussion Co-learning is a more effective approach than active learning in obtaining feedback from human annotators for optimizing the language model. With sufficiently labeled samples, a language model such as BioBERT outperformed large language models.Conclusion In both maternal and pediatric pharmacology research, the absence of PK evidence was notably more frequent compared to PE and CT. This significant finding calls for more pharmacokinetics studies within pediatric and maternal patient populations.
