ABSTRACT Introduction The reliability of real‐world evidence (RWE) depends on data accuracy and secondary users' ability to evaluate how data are generated and transformed. Although recent frameworks emphasize provenance, transparency, and auditability, their practical assessment in routine hospital settings remains unclear. This study examines the practical evaluability of real‐world data (RWD) and explores operational boundaries limiting data quality assessment within learning health systems. Methods An empirical case study was conducted at a tertiary‐care university hospital in Japan. Data extracted from electronic health records and a clinical data warehouse were compared using 15 quality indicators derived from the Data Quality Management Guidebook of the Digital Agency of Japan (based on ISO/IEC 25012). Evaluability was assessed using a two‐step framework comprising empirical verification and contextual explanation from the perspective of research support personnel operating under routine institutional access permissions. Results Evaluability varied substantially across clinical domains. Diagnosis records were generally verifiable and explainable, whereas medication‐related domains contained observations that could be verified but not fully explained by secondary users. Indicators including currentness and efficiency faced additional operational constraints, revealing boundaries between observable data characteristics and available explanatory information. Conclusions Practical RWD use depends not only on observable data characteristics but also on the accessibility of information needed to interpret them. Evaluability boundaries reflect operational constraints rather than poor data quality. Rendering these boundaries visible may support more transparent and accountable RWD governance and inform future improvements through system design and data governance practices.
Introduction:The Syphilis Point of care Rapid testing and Immediate Treatment Evaluation (SPRITE) is a collaborative, community-based initiative that strengthens syphilis prevention and control in Ontario, Canada through flexible outreach nursing and point-of-care testing (POCT) and immediate treatment. Seven public health units (PHUs) partnered with 86 community-based organizations (CBOs) to deliver syphilis/HIV POCTs in underserved settings. This study evaluated a real-time knowledge exchange network that supported syphilis POCT implementation and capacity-building across PHUs and CBOs in Ontario. Methods:The evaluation was a mixed-methods case study using surveys and interviews focusing on the community of practice (CoP), community engagement, and knowledge mobilization. It considers SPRITE CoP activities from beginning of implementation (mid 2023) to the Fall of 2025. Results:The CoP, consisting of public health nurses, researchers, and knowledge specialists, supported engagement, co-development, and peer learning. Members shared lessons learned, built partnerships, and supported each other in navigating challenges. Flexibility in funding allowed innovation by CoP members in their CBO engagement efforts but also created some uncertainty and possible confusion. CoP members were overall satisfied with CoP functioning, and CBOs surveyed appeared knowledgeable and engaged in SPRITE. Discussion:The CoP model, while distinct from traditional public health CoPs, proved effective in fostering engagement, resource development, and knowledge mobilization. Lessons learned and tools developed provide guidance for future POCT efforts. Improvements should include clearer goals, stronger facilitation, and structured engagement strategies. Addressing funding stability, particularly for POCT services and CoP management roles, is essential. Strengthening CBO partnerships through incentives and bi-directional information sharing will further enhance POCT and syphilis prevention. This evaluation underscores the importance of flexible yet accountable approaches to knowledge mobilization and POCT implementation in diverse public health contexts. Continued investment in collaborative networks like SPRITE remains critical to advancing syphilis elimination efforts across Ontario.
Introduction:Sustaining interventions implemented as part of pragmatic clinical trials requires proactive planning, but how to approach this process remains understudied. The Non-pharmacological Options in Hospital-based and Rehabilitation pain Management stepped-wedge cluster-randomized pragmatic trial tested an electronic health record-based educational bundle (known as the Healing After Surgery [HAS] initiative) that encouraged the incorporation of non-pharmacological pain care in perioperative pain management. The HAS initiative was implemented in multiple surgical practices and hospital sites. One year prior to trial completion, a sustainment committee was established to support the transition of intervention components to clinical ownership and to support posttrial sustainment of the HAS initiative. Methods:We reviewed meeting minutes from biweekly sustainment committee meetings and conducted debriefing sessions with committee members (informed by the Clinical Sustainability Assessment Tool) to identify sustainment strategies and challenges, which were then organized into meaningful themes. Results:The following six themes emerged from meeting minutes and debriefings: (1) automation of low-touch components, (2) reliance on internal champions, (3) intentional handoff communication, (4) institutional attention, (5) relaxed fidelity, and (6) continuous evaluation. Together, they highlight the importance of early, structured planning and adaptable implementation strategies. Conclusions:Sustainment of the HAS initiative required extensive communication, adaptation, and stakeholder engagement across diverse institutional contexts. Proactive sustainment planning during trial design may help ensure interventions are successfully integrated into routine clinical practice posttrial.
ABSTRACT Introduction Learning Health Systems (LHS) offer a pathway to continuously improve care by integrating data and lived experience, yet few practical examples demonstrate how LHS can be operationalized in hospital settings in ways that embed the voices of patients and families. This manuscript describes how patient experience (PX) data are integrated into the design of a new model of care (MoC) that responds to both system demands and the realities of patients and families. Methods Researchers partnered with clinical leaders to integrate patient and family voices into the design of a new MoC using PX survey data, interviews with patients and families admitted to medicine units, and naturalistic walk‐throughs of each medicine unit. Data collection explored care priorities, interactions with staff, physical environments, and reflections on positive and challenging aspects of care. Analysis followed a three‐step iterative process: unit‐level synthesis, cross‐unit thematic comparison, and data integration. Results In total, 390 patients participated in the PX survey and 19 patients/families participated in interviews. Patients situated their experiences within a health care system under strain, highlighting both strengths and gaps. Many expressed appreciation for nurses' compassion, but emotional support for fears, anxieties, and worries was inconsistent, particularly when staff were overstretched or transient. Long wait times for transport, procedures, or basic care needs were reported as sources of anxiety and diminished dignity. Naturalistic walk‐throughs documented how crowding, noise, lighting, and wayfinding shaped patients' comfort and privacy. Conclusions This work provides a practical example of how an LHS approach can be operationalized by embedding the voices of patients and families into care redesign. Findings from the PX survey and interviews are directly shaping a new MoC at Trillium Health Partners (THP) that is evidence‐informed and co‐designed with patients and families, providing a practical example of how PX data can drive continuous, system‐level learning.
ABSTRACT Introduction The UK National Institute for Health and Care Excellence (NICE) produce guidelines that provide evidence‐based recommendations to support clinical care across England and Wales, but remain available in unstructured natural language form. Converting these guidelines into computable, logically coherent representations is an active area of research yet existing approaches typically focus on individual diseases, require substantial manual encoding, and do not scale. Recent advances in large language models offer an opportunity to automate much of this translation process. Methods We present an end‐to‐end approach that automatically converts textual clinical guidelines into an executable model capable of generating explainable patient‐specific recommendations. Our approach uses a stepwise LLM‐based transformation with in‐context examples that can be customized to the guideline of your choice. Each step generates human‐inspectable intermediate artifacts, ensuring full transparency and modifiability. We apply the approach to both pancreatic and lung cancer NICE guidelines and use expert human review to assess the alignment of the produced rules as well as evaluating the executable model over 20 pancreatic cancer patient vignettes. Results Human experts review demonstrated strong alignment between the natural language guidelines and the generated executable models, with the majority of guideline recommendations translated correctly. Most discrepancies involved partial omissions of specific details rather than incorrect logic, and instances of hallucinated or fundamentally incorrect rules were rare. When executed on the vignettes, the resulting executable models produced patient‐specific recommendations with an F1 score of 82.5%. Conclusion This work demonstrates that LLMs can be used to automatically transform natural language NICE guidelines into interpretable and executable models. The models preserve guideline structure, allow transparent inspection and modification, and can be executed to generate patient‐specific recommendations. Our findings highlight the feasibility of automated guideline generation, opening the door to scalable computable guidelines.
Introduction:Learning communities need access to resources to support efforts in building and sustaining learning health systems. This experience report details our development and initial assessments of the Learning Health System (LHS) Toolkit, a menu of resources purpose-designed to support users to build more proactive, responsive, and equitable systems of care by developing, implementing and sustaining LHSs. Methods:Toolkit development began in 2022, and the current version was completed in 2025. To optimize knowledge translation, we structured our methods to align with the phases of the Knowledge to Action (KTA) Model, including (1) Knowledge Inquiry; (2) Synthesis and Creation; (3) Knowledge Selection; (4) Adapting Knowledge to Local Context; (5) Assessing Barriers and Facilitators to Knowledge Use; (6) Selecting, Tailoring, and Implementing the Intervention, and (7) Monitoring Knowledge Use. Results:Development of the web-based toolkit and vetting of its contents were achieved over several phases. Feedback from toolkit users was positive overall and was integral to its development and refinement. Users from 72 countries have accessed the toolkit, although most engagement has occurred in high-income countries, chiefly the US, UK, Türkiye, Australia, Canada, The Netherlands, and New Zealand. Conclusions:The toolkit is uniquely situated to support learning communities. It includes a dynamic design, is regularly updated, and is accessible free of cost. Additionally, unlike a literature search, the toolkit is developed with usability in mind and includes tools beyond peer-reviewed literature, curated by LHS experts. The next steps are to develop and action a dissemination plan based on implementation science principles to increase the reach and adoption of the toolkit.
Ambient artificial intelligence (AI) scribes are systems that automatically generate clinical documentation from clinician-patient conversations and are being deployed at accelerating pace across US health systems. Early evaluations report reduced documentation burden, improved clinician well-being, and perceived efficiency gains, reinforcing a narrative of inevitability. Yet this frontline framing understates a more consequential issue: ambient scribes outsource the "first mile" of clinical documentation, thereby reshaping the production of clinical data and the learning health systems (LHSs) that depend on documentation as foundational infrastructure. This paper argues that ambient AI scribes should be understood not merely as workflow tools, but as emerging infrastructure that will materially shape the capacity and capabilities of LHSs. Drawing on infrastructure studies and LHS frameworks, we conceptualize clinical documentation as the epistemic substrate through which encounters are translated into analyzable data that power quality measurement, predictive modeling, clinical decision support, and institutional learning. When this translation is algorithmically mediated by proprietary systems, design choices, training data, and integration pathways can introduce systematic documentation errors that propagate downstream, often invisibly, through analytic pipelines. Synthesizing emerging evidence, we highlight risks including hallucinated clinical details, omission of safety-critical information, and differential performance across patient populations with diverse accents or speech patterns. These risks mirror classic infrastructural properties described by Star: embeddedness, dependence on the installed base, wide propagation, and visibility primarily upon breakdown. From this perspective, ambient scribes may quietly reshape documentation norms, data quality, and learning trajectories well before downstream effects are routinely assessed. We conclude by outlining a governance agenda grounded in LHS principles: documentation-quality metrics, drift monitoring, equity-focused evaluation, transparency, and multi-stakeholder stewardship. Without such oversight, ambient AI scribes risk stabilizing an infrastructural layer that delivers short-term relief while eroding the long-term integrity, equity, and trustworthiness of learning health systems.
Introduction:The Learning Health System (LHS) concept is gaining traction in high-income countries, yet its implementation in low- and middle-income countries (LMICs) remains limited with few initiatives offering practical strategies tailored to these contexts. The Leading a Learning Health System (LLHS) Program, grounded in LHS principles, aimed to build leadership capacities among health professionals and leaders from five Asia-Pacific LMICs. This study explored participants' learning experiences and applications in their context. Methods:A mixed-method longitudinal study was conducted (13-months), informed by social constructivism and Kirkpatrick's evaluation framework. Surveys and semi-structured interviews were conducted after the intensive course (Survey A and Interview A) and 1 year later (Survey B and Interview B). Quantitative data were analyzed descriptively, and qualitative data were analyzed through inductive content analysis. Results:Thirteen participants (13/20, 65%) completed Survey A. All (100%) rated the program and cross-country network as "high value." Eleven participants (11/20, 55%) completed Survey B. All participants reported applying program learnings regularly on most days (6/11, 55%) or most weeks (5/11, 45%). Nine participants (82%) reported influencing team culture and five (45%) reported influencing organizational change. Qualitative analysis of 20 Interview A and 10 Interview B identified four themes: (1) developing a systems leadership identity, (2) translating leadership intent into practice, (3) contextual challenges to applying learnings, (4) resilience via adaptive leadership capacity and programmatic support. Conclusion:This study demonstrates how a longitudinal leadership program based on the LHS framework supported LMIC health professionals to develop systems leadership identity, foster learning culture, and cultivate adaptive, distributed leadership to build resilience in resource-limited settings. The variability of health information systems in LMICs should not be seen as a barrier to implementing LHS, as improvement can begin with the effective use of locally available data. These findings can inform future capacity-building initiatives to advance LHSs in LMICs.
Introduction:Physical function (PF) is critical to quality of life and healthcare value, especially for older adults following hospitalization. Monitoring PF supports recovery, reduces adverse events, and improves care transitions. Despite the potential of electronic health records (EHRs) enabling systematic PF tracking, such data are rarely captured consistently. Here we examine the availability of PF-related data in EHRs for patients transitioning from hospital to homecare in a large health system, highlighting challenges and offering recommendations. Methods:We assessed availability of elements previously identified important to PF measurement from a single healthcare system. Working with Johns Hopkins Health System informatics and homecare leaders, we determined which recommended elements were captured in the EHR and which were feasible to extract within our resource constraints. We then requested an extraction of a refined data set for adult patients with a hospital admission between July 2016 and March 2021. After validation, data were securely transferred to University of Utah Health. Results:Data from 21 702 patients were included. Of 27 desired elements, 17 were available and successfully extracted. Individual elements were marked "present" if documented at least once during admission, or "missing" if absent. Administrative data had low missingness, although missingness for assessments of cognition and mobility performance in hospital was over 65%, and assessments of PF capacity in home health were missing in over 80% of patients. However, 81.7% of those receiving home health rehabilitation had the expected mobility measure. Overall, 73% of patients had at least 75% of the extracted data elements. Conclusions:Assembling a comprehensive view of PF across a care transition using EHR data proved highly challenging. Our recommendations address data element identification, generation and storage; data extraction, cleaning, and validation; interoperability across care settings; adequate resources to manage complex data; and prospective infrastructure development.
ABSTRACT Introduction Unique challenges of data quality, standardization, and interoperability face non‐traditional sources of real world data (RWD), those outside of the common RWD source, the electronic health record (EHR). The experience of our clinical, research, and community group in addressing the endangered school‐based physical fitness testing (SB‐PFT) data set may be useful as a paradigm for a variety of nontraditional instances of health‐related RWD. Until recently, SB‐PFT was mandated annually within public schools in ~60% of U.S. children and adolescents representing one of the most robust and inclusive data sets available in youth. When implemented rigorously, SB‐PFT produced data that improved pediatric health and student learning. Driven by perceived and real gaps in data quality, inconsistent and insensitive school‐site implementation, and uneven evidence for health‐relevance, calls for revamping (and, in some cases, ending) school testing are growing. Methods To preserve and enhance this unique resource, SB‐PFT must incorporate emerging science and technology. We describe here the rationale, structure, and process involved in our building an HL7‐associated Domain Analysis Model for SB‐PFT. Our team consisted of clinicians, exercise scientists, school personnel, publish health officials, parents and students. The structure of the DAM was facilitated by data standard experts and skilled Unified Modeling Language professionals. Administrative support was rendered by our institution's NIH‐funded Clinical Translational Science Award. Results The DAM will serve as a necessary first step in integrating Fast Healthcare Interoperability Resources (FHIR) and Learning Health Systems (LHS) models essential to realize the full potential of SB‐PFT data in advancing child health. Conclusion The experience presented here can serve as a roadmap for addressing critical issues of data standardization and RWE generation in nontraditional instances of RWD. The DAM can be deployed in SB‐PFT data exchange in real world settings.
Introduction:The allocation and timing for kidney matching and transplantation among incompatible donor-recipient pairs in the Paired Kidney Exchange face significant operational challenges of managing an uncertain waitlist. The time for dialysis of patients waiting for biologically compatible kidneys follows the concept of a stochastic queueing system. This study embodies the Learning Health System approach by integrating real-world data and mathematical modeling to generate actionable insights that guide decision-making. Methods:This work employs an M / M / 1 model with Poisson arrivals and exponentially distributed service times for incompatible donor-recipient pairs in a single transplant facility. Inter-arrival and service times are used to derive queue length and waiting-time measures to characterize system congestion and instability, with performance measures evaluated as indicative approximations analyzing the corresponding metrics. Results:The system is persistently overloaded, with arrival rates exceeding service rates ( λ > μ ; ρ > 1), causing prolonged waiting times. Analytical and Hutson-B bootstrap confidence interval analysis shows congestion persists across plausible variations, and sensitivity analysis indicates moderate perturbations do not alleviate overload. Regression and correlation analyses indicate that delays are associated with arrival pressure and biologically imposed constraints, supporting compatibility-constrained matching as the primary operational challenge. Patients experience uncertain, extended waiting periods, reflecting stochastic instability in biologically constrained queueing systems. Conclusions:Transplant service management requires systematic reforms, as clinical matching and allocation delays hinder timely care. Establishing a national transplant registry, integrated real-time queue monitoring, decentralized planning with expansion of transplant centers and improved inter-center coordination for paired exchange and increasing its cycle lengths are essential to ensure efficient service delivery and informed healthcare policy. These interventions create a continuous feedback loop where operational data informs queueing-theoretic analysis, informs policy, system redesign, and ultimately improves patient access and outcomes, demonstrating how data-driven learning optimizes complex healthcare services.
ABSTRACT Introduction Learning Health Systems (LHSs) depend on the ability to effectively evaluate programs and translate data into practice. Within the Veterans Affairs (VA) health system, formal mandates require programmatic decision‐making to be data‐driven; however, opportunities for staff to develop applied program evaluation skills that support LHS learning cycles remain limited. Methods To address this need, the synchronous virtual Evaluation Bootcamp Training (EBcT) was developed to equip learners with practical, applied evaluation skills. This manuscript describes the design rationale, structure, and core concepts of the training, emphasizing design decisions intended to support evaluation capacity building within a large and complex health system. Results EBcT was delivered over 4 days across 2 training cohorts (8 learners in 4 project teams from Cohort 1; 12 learners in 5 project teams from Cohort 2). Project teams addressed a range of clinical and operational initiatives within VA. Learners represented diverse professional roles, including lead evaluators, clinical subject matter experts, project managers, operational leaders, and analysts. Learners reported high satisfaction throughout the training, with full attendance at application sessions and strong participation in didactic components. A majority reported meeting predefined evaluation learning targets at the end of the training. Conclusion Findings suggest that structured, team‐based training models such as EBcT can strengthen evaluation capacity and support LHS functioning by equipping project teams to engage in repeated cycles of measurement, learning, and performance improvement.
ABSTRACT Objective To analyze the barriers, enablers, and strategies affecting the implementation of learning health systems (LHS) projects across individual (micro), organizational (meso), and system (macro) levels within Australian healthcare settings Methods Semistructured interviews were conducted with 26 clinicians undertaking LHS projects in their organizations, as part of a LHS Academy fellowship program. Interviews were completed at two discrete time points: early in project implementation and again at project completion. They explored fellows' experiences with implementing LHS projects in real‐world organizational contexts. Data were analyzed using a hybrid deductive‐inductive approach guided by a multilevel ecological codebook developed by Shaw et al. Results Participants' early interviews highlighted limited confidence, emerging leadership skills, and uncertainty in navigating implementation challenges, while later interviews described greater confidence, stronger leadership capability, and more refined adaptive strategies at the micro level. Organizational influences were described at both time points as slow and uneven, though later interviews included examples of small cultural shifts and early collaborative efforts alongside persistent constraints such as limited resources, siloed cultures, and inadequate digital infrastructure. At the system level, barriers including regulatory complexity, policy misalignment, and limited funding were reported consistently across both interviews and were perceived as largely unchanged. Conclusions Building individual capability was necessary but insufficient for successful LHS project implementation. Alignment across micro‐, meso‐, and macrolevels is critical, requiring parallel investment in organizational readiness, digital infrastructure, and policy reform. Fellowship programs may support individual change and foster proof‐of‐concept projects, but scaling impact demands coordinated, multilevel strategies.
Introduction:Person-centered care planning (PCCP) involves active collaboration between people seeking care, clinical teams, and others to co-create longitudinal treatment plans. It is a crucial part of care for people with multiple chronic conditions (MCCs) and other complex needs. Despite the widespread acceptance of the concept, the use of PCCP in the US is variable. Many effective models of PCCP exist but uptake has been limited. Objective:We sought to identify and assess current models and approaches for PCCP by performing a multi-component environmental scan. Methods:We conducted targeted literature reviews based on streamlined systematic review methods and qualitative interviews with key informants, identified for their relevant expertise and frontline knowledge of PCCP models. Results:In all, 966 abstracts and 187 full articles met review criteria. We identified 40 models with elements fitting into 7 categories of how they differed from usual care (adding People/Roles, innovative Payment/Incentives, novel Technology, decision support Tools, Services, Functions, and Changing Focus). Most were multi-component models with evidence of effectiveness in outcomes that address the quintuple aim. Barriers to adoption and integration across sectors were primarily lack of time and appropriate payment mechanisms. A small set of measures for PCCP was identified that focused primarily on patient experience and goal setting. Similarly, a small number of PCCP models connected with social services organizations to address health-related social needs and highlighted challenges with data sharing and payment as well as growing collaborations in community care hubs to address social needs. KI interviews echoed and deepened our understanding of these findings by contextualizing experiences with barriers and factors facilitating the delivery of PCCP. Discussion:PCCP is a key aspect of high-quality care for people with MCCs; uptake is limited with barriers related to time and resources. Facilitators include alignment with healthcare system objectives and practices, and ready-to-deploy resources that enable coordinated PCCP care across providers. To promote greater adoption of PCCP concerted efforts are needed to share implementation details, success stories, and build the case for PCCP with patients, their families, and healthcare system leaders.
Introduction:Artificial intelligence is increasingly embedded in healthcare delivery, yet existing Learning Health System (LHS) models do not fully account for the lifecycle management and continuous assurance requirements of AI systems. This gap limits health systems' ability to safely and sustainably integrate AI as a learning component of care. Methods:We conducted a conceptual system modeling investigation grounded in LHS theory and contemporary AI governance frameworks. Through structured theoretical integration, we aligned the classical LHS learning cycle with an action-oriented AI lifecycle and five continuous assurance dimensions, developing a unified framework to support operational implementation within health systems. Results:The resulting Health AI Learning and Oversight (HALO) model specifies how AI functions as a dynamic knowledge artifact within an LHS. Application of the model illustrates how integrating lifecycle stages and continuous assurance instantiates iterative learning loops, enables adaptive governance, and supports operational lifecycle management, including ongoing monitoring of performance, safety, equity, transparency, and security across clinical environments. Conclusions:By extending LHS theory to incorporate AI lifecycle and assurance requirements explicitly, the HALO framework operationalizes continuous learning and oversight for AI-enabled health systems. This model provides a foundation for designing, governing, and sustaining responsible and adaptive AI deployment as healthcare environments evolve.
ABSTRACT Introduction Active post‐marketing surveillance of prescribing behavior of high‐risk drugs may provide early warning of unforeseen issues in a population, yet analysis approaches for surveillance using real‐world data are underdeveloped. This paper evaluates a modified statistical process control (SPC) method for surveillance of risk minimization measures derived from administrative claims data. The approach detects changes in population‐level prescribing behaviors and informs investigators of the timing and nature of any detected changes. Methods We investigated prescription drug claims for tapentadol extended release (ER) from the Colorado All‐Payers Claims Database (2012–2019). The cohort was 2702 unique patients receiving their first prescriptions of tapentadol ER (an opiate with an FDA Risk Evaluation and Mitigation Strategy). The risk minimization measures were the prescribing rate and the proportion of prescriptions with appropriate dosing. A statistical model was fitted to data from January–December 2012 to establish a stable baseline and then updated biweekly through June 2019. We conducted surveillance with our modified SPC method and a classical SPC method, controlling the false alarm rate to 0.005 for each, and compared how detections aligned with external policy actions. Results Our method detected two periods of unusual prescribing behavior beginning in March 2015 (p < 0.005) and April 2016 (p < 0.005). The classical method detected a single period in October 2016 (p < 0.005). Our detections aligned with risk minimization activities in Colorado; the classical method aligned with only the second activity, 6 months later. Conclusion Our modified SPC method, which monitors model misspecification rather than raw prescribing data, detects more periods of changing behavior that better temporally align with risk minimization actions. This tool may be useful for regulatory agencies to independently monitor for emerging risks and prescribing trends to complement REMS assessments.
Introduction:Stakeholder engagement is a core element of a learning health system (LHS). Meaningful engagement of stakeholders can improve learning within a health system by informing program development and service delivery and by increasing buy-in and sustainability of evidence-based practices. While health systems recognize the value of stakeholder engagement, conducting stakeholder engagement activities requires time, resources, and the support of health system leadership. The objective of this pilot study was to obtain preliminary data on the value of stakeholder engagement from the perspective of rehabilitation directors. Methods:We conducted a general qualitative study utilizing 90-min focus groups. The main outcomes were domains identified from analysis of the focus group transcripts. Results:Six individuals from 4 health systems participated in one of two focus groups. Seven domains were identified: (1) how stakeholder engagement supports organizational mission and priorities; (2) types of stakeholders working with health systems; (3) past stakeholder engagement experiences; (4) engagement with stakeholders with lived experiences; (5) sustaining stakeholder relationships; (6) challenges with stakeholder engagement; and (7) future stakeholder engagement goals. Conclusions:This qualitative pilot study of the perspectives of rehabilitation LHS leaders on stakeholder engagement provides preliminary data that can be used to inform future research. While we found consistent themes recognizing the value of stakeholder engagement and participation in stakeholder engagement, respondents noted challenges in engaging stakeholders and using the information gained to effect change.
ABSTRACT Background The VA Office of Research and Development (ORD) has re‐organized such that certain high priority health areas, such as suicide prevention, are managed using an Actively Managed Portfolio (AMP) model. One key capability of AMPs is to establish priorities for the portfolio; here, we describe the process and outcome of the priority setting process for the Suicide Prevention AMP (SP‐AMP). Method The SP‐AMP utilized a three‐phase process for determining suicide prevention research priorities. Team members initially identified existing priority statements from organizations invested in suicide prevention, and in a second phase, obtained feedback from internal VA and non‐VA partners as well as Veterans on the perceived importance of these domains. In the final phase, an executive steering committee (ESC) considered the qualitative and quantitative data for five potential areas for prioritization. Results The ESC selected “lethal means safety (LMS) approaches to suicide prevention” as the priority area for 2025. Conclusions SP‐AMP team solicited feedback from multiple relevant partners in suicide prevention. “LMS approaches to suicide prevention” was selected as the key immediate priority because advances could lead to immediate impact in suicide prevention. Other candidate domains will be highlighted by the SP‐AMP in upcoming funding cycles.
ABSTRACT Background To assess acceptability, feasibility, and effectiveness of incorporating individualized risk prediction into clinical assessment, decision making, and communication of risk of type 2 diabetes, with and without preventive interventions, in patients with prediabetes. Methods We integrated a prediction model into the clinical workflow at a U.S. health care organization. We conducted patient and provider focus groups and pre‐ and post‐dissemination surveys among 2500 patients with prediabetes who had primary care visits between May 2018 and December 2019. We compared rates of progression to type 2 diabetes at 3 years between the intervention group and a propensity score‐matched cohort of patients who received usual care. Results Prior to implementing the predictive model, 41.6% of providers and 63.8% of patients felt confident or very confident in their ability to estimate the risk of progression to diabetes for individual patients. After personalized risk information was made available, this increased to 92.8% for providers and 66.9% for patients. People with prediabetes who had a primary care visit where their care team had access to personal risk of developing type 2 diabetes assessed by the EHR‐based prediction model were significantly less likely to progress to diabetes within 3 years, compared to a propensity‐score‐matched cohort who received usual care in the same health system without individualized risk estimates (19.5% vs. 27.6%, p = 0.042). Conclusions Used at the point of care during a primary care visit, the EHR‐based diabetes risk calculator helped providers prioritize patients for diabetes preventive interventions, facilitated communication, and improved health outcomes among patients with prediabetes.
Objectives:Embedding systematic, structured data extraction within electronic health records (EHR) is vital for improved real-time insights into care delivery. This study evaluates the feasibility of using large language models (LLMs) to extract structured advance care planning (ACP) information from unstructured Goals of Care (GoC) clinical notes in the EHR. Materials and Methods:A sample of 100 de-identified GoC notes was manually annotated by clinicians across four ACP categories: Patient Priorities, Code Status, Decision Maker, and Documentation. Two LLMs (Mistral 24.07 and LLaMA 3.1) were prompted to extract structured outputs without domain-specific fine-tuning. Model outputs were compared to human annotations using cosine similarity of BioBERT embeddings. Results:Mistral 24.07 achieved high semantic similarity in Code Status (0.814), Documentation (0.781), and Patient Priorities (0.770), but lower alignment in Decision Maker (0.609). Conclusions:LLMs can effectively extract structured ACP information, particularly in well-documented categories, suggesting potential for scalable, data-driven feedback loops that improve the provision of care. However, accuracy challenges remain, and further refinement is needed for nuanced qualitative content categories.
