ABSTRACT Introduction Real‐world health data (RWD), typically derived from electronic health records, administrative claims, patient registries, wearable devices, and other non‐experimental sources, are increasingly used to generate real‐world evidence (RWE) in public health and clinical medicine. While RWD offers multiple advantages over traditional randomized clinical trials (RCTs) in cost, feasibility, speed, and generalizability, methodological pitfalls can undermine the reproducibility and validity of findings based on RWD analysis. Methods In this manuscript, we highlight key methodological challenges in RWD analysis and illustrate them with published case studies. We then review published methodological solutions, including target trial emulation, propensity score adjustment, instrumental variable approaches, and inverse probability of sampling weighting, that can improve the validity and generalizability of empirical results derived from RWD. Results We contend that awareness of RWD limitations is not a justification for avoiding its use, but a mandate to use it more carefully. Conclusions Pre‐specifying sampling and analytic plans, applying robust inferential methods, and continuing to invest in novel bias‐reduction methods are essential to ensuring that RWE fulfills its promise for public health research and policy.
ABSTRACT Background Healthcare Education and Research Training (HEART) is a cocurricular, engaged learning initiative that supported undergraduate students in acquiring and applying healthcare research skills. The project was guided by a learning health system (LHS)‐aligned model focused on the collection, synthesis, and rapid translation of patient‐reported experience data into actionable recommendations. In the 2023/2024 academic year, a cohort of 28 students completed the pilot project to report LHS‐based recommendations to healthcare directors. This paper reports the qualitative evaluation conducted in August 2024 following the pilot to assess the initiative's implementation, participant experience, and healthcare system impacts. Methods Virtual, semi‐structured interviews were conducted with students who participated in HEART and health system directors from a provincial health authority. Interview guides were developed by the research team in collaboration with Patient Partners. Inductive content analysis (ICA) was used to categorize information significant to the research questions. Final themes were established using a team‐based approach. Results Student interviews indicated that participants found HEART to be a meaningful learning experience that strengthened their understanding of the healthcare system and patient experiences and helped them build applied research skills. Health system directors also expressed strong satisfaction with the project, noting that student‐collected qualitative data identified service gaps, supported patient and community engagement, and contributed to measurable care improvements in at least one clinic. Suggested areas for project improvement included adjusting length of training sessions, increasing clarity around expectations and timelines, streamlining final reporting, and refining the patient interview process to gather more detailed information about specific patient experiences. Conclusion While various opportunities for improvement were identified, overall, HEART was determined to be a valuable opportunity for developing student healthcare research skills and was of direct benefit to health system partners as an LHS‐aligned initiative providing actionable data used to identify healthcare gaps and improve patient care. Feedback and recommendations drawn from the evaluation were used to adapt and improve HEART for the following academic year.
ABSTRACT Introduction Our group developed a calculator to predict individualized likelihood of cesarean delivery for patients undergoing labor induction. We aimed to determine whether nudge theory informed EHR integration of the cesarean risk calculator would increase (1) calculator utilization, and (2) documentation of patient counseling on the result. Methods This was a prospective cohort comparing 6 months pre (10/2022–4/2023) versus 6 months post EHR‐release of the cesarean risk calculator (4/2023–10/2023) in two labor units. Prior to EHR‐integration, Site #1 had a 2018 focused effort towards calculator website use, while Site #2 had no previous use. EHR‐integration was automated to recognize eligible patients and allowed clinicians to confirm and utilize EHR variables for score automation, visible on electronic display. Counseling recommendations, specific to the patient's cesarean risk level, were then available for insertion into an EHR‐note. The primary outcomes were (1) calculator utilization measured as a score generated in the EHR and (2) risk counseling defined as documentation of score discussion with the patient. The primary outcomes were also analyzed stratified by site. Results A total of 2018 patients met eligibility criteria for the cesarean risk calculator (pre:998; post:1010). Clinician calculator utilization significantly increased POST‐EHR implementation from 13.4% to 77.4% (p < 0.001). Site #2, without a prior effort towards calculator use, had greater increase in use with EHR integration compared to Site #1 (Site#1 pre: 28.8% vs. post: 62.3%, p < 0.001; Site#2 pre: 0.4% vs. post: 91.8%, p < 0.001). Documentation of patient counseling also increased overall (4.0% vs. 9.9%, p < 0.001); when stratified by site, this difference was only seen at Site #2. Conclusion Behaviorally designed EHR integration of a cesarean risk calculator for labor induction was associated with significant increases in calculator utilization and documentation of patient counseling. This model could be applied to other evidence‐based health‐related risk calculators.
ABSTRACT Introduction Health systems are expanding access to artificial intelligence (AI), yet frontline integration into teaching, clinical workflows, and research practice remains uneven. We frame this gap as the “AI embedding deficit,” reflecting limited organizational pathways that enable frontline professionals to translate local needs into legitimate, supported, and sustainable AI‐enabled tools and workflows. This Experience Report describes a prototype intervention mechanism implemented at an academic nursing school: a rapid‐cycle Nursing‐AI Challenge followed by a year‐round Nursing‐AI Studio (NAIS) designed to stabilize and scale embedding. Methods We implemented the Nursing‐AI Challenge as a structured, time‐bounded mechanism to solicit real‐world challenges from nursing faculty, translate and curate submissions through an interdisciplinary facilitation role, match challenges to AI trainees and mentors, and support rapid proposal development culminating in a public showcase. We then developed NAIS to address continuity and embedding barriers revealed in the Challenge, incorporating continuous intake, environmental scanning, supported co‐development, year‐round coordination, formalized interdisciplinary partnerships, and a strategic memorandum of understanding with the Chief Information Officer's office to support legitimate integration pathways. Results In the inaugural Challenge cycle (Spring 2025), nursing faculty identified 17 challenges spanning education, clinical practice, and research, and students in AI‐related fields submitted eight proposals, followed by team formation, co‐design of the AI prototypes, and a public showcase event. Implementation reflections identified key sustainability barriers including semester‐bounded participation, summer momentum gaps, and intellectual property ambiguity. NAIS was established (Fall 2025 onward) to mitigate these barriers and support ongoing embedding. In early Spring 2026, NAIS launched “Teaching with AI,” a clinical faculty innovation initiative piloting an AI‐powered teaching tool fully deployed within the university's information technology infrastructure. Conclusions A Challenge‐to‐Studio pathway may function as a practical prototype mechanism for learning health systems seeking to reduce embedding barriers and convert decentralized frontline innovation into durable, governable, and evaluable practice change.
ABSTRACT Introduction The reliability of real‐world evidence (RWE) depends on data accuracy and secondary users' ability to evaluate how data are generated and transformed. Although recent frameworks emphasize provenance, transparency, and auditability, their practical assessment in routine hospital settings remains unclear. This study examines the practical evaluability of real‐world data (RWD) and explores operational boundaries limiting data quality assessment within learning health systems. Methods An empirical case study was conducted at a tertiary‐care university hospital in Japan. Data extracted from electronic health records and a clinical data warehouse were compared using 15 quality indicators derived from the Data Quality Management Guidebook of the Digital Agency of Japan (based on ISO/IEC 25012). Evaluability was assessed using a two‐step framework comprising empirical verification and contextual explanation from the perspective of research support personnel operating under routine institutional access permissions. Results Evaluability varied substantially across clinical domains. Diagnosis records were generally verifiable and explainable, whereas medication‐related domains contained observations that could be verified but not fully explained by secondary users. Indicators including currentness and efficiency faced additional operational constraints, revealing boundaries between observable data characteristics and available explanatory information. Conclusions Practical RWD use depends not only on observable data characteristics but also on the accessibility of information needed to interpret them. Evaluability boundaries reflect operational constraints rather than poor data quality. Rendering these boundaries visible may support more transparent and accountable RWD governance and inform future improvements through system design and data governance practices.
