Introduction:We report on using electronic health records (EHRs) and other health information technology (IT) (eg, REDCap, Excel, and population-health tools) for tracking patients and managing interventions to improve colorectal screening (CRC) among primary care practices who participated in the National Cancer Institute's Accelerating Colorectal Cancer Screening and Follow-up through Implementation Science (ACCSIS) program. Methods:We conducted semi-structured, recorded interviews with staff from 7 ACCSIS Research Projects (RPs). Using the interview notes, we conducted content analysis to report on the characteristics of the EHR systems and health IT, and thematic analysis to identify key concepts related to the ability to capture and monitor data for CRC screening. Results:RPs used different data capture models to support EHRs and health IT: (1) centralized data capture models from projects or third-party services; or (2) direct data capture models, relying on features and functions within commercial EHRs. Respondents reported challenges to using EHRs and health IT, including generating patient reports to track interventions, working across EHR and research platforms because of lack of interoperability, and training for clinic staff on EHR and research platforms. Discussion:RPs would benefit from more streamlined data capture and reporting for managing CRC screening in primary care. Efforts reportedly fell onto staff who could have benefited from training around data handling and EHR-specific navigation. Conclusions:RPs experienced challenges in leveraging data capture models for EHR and health IT data management. Our research calls for technical capabilities that promote more efficient data capture and reporting, as well as greater capacity building among clinic staff.

OBJECTIVES:Advances in informatics research come from academic, nonprofit, and for-profit industry organizations, and from academic-industry partnerships. While scientific studies of commercial products may offer critical lessons for the field, manuscripts authored by industry scientists are sometimes categorically rejected. We review historical context, community perceptions, and guidelines on informatics authorship.PROCESS:We convened an expert panel at the American Medical Informatics Association 2022 Annual Symposium to explore the role of industry in informatics research and authorship with community input. The panel summarized session themes and prepared recommendations.CONCLUSIONS:Authorship for informatics research, regardless of affiliation, should be determined by International Committee of Medical Journal Editors uniform requirements for authorship. All authors meeting criteria should be included, and categorical rejection based on author affiliation is unethical. Informatics research should be evaluated based on its scientific rigor; all sources of bias and conflicts of interest should be addressed through disclosure and, when possible, methodological mitigation.

Background: Hospital settings provide a unique opportunity to screen for interpersonal violence (IPV) and sexual assault (SA) yet often lack health IT solutions for generating reliable and valid medico-legal documentation via forensic reports. Objectives: The objective of the project was to evaluate a pilot, technology “tool” for documenting cases of IPV and SA that could support forensic nurse examiners and related stakeholders in generating high quality documentation and coordinating victim support services. Methods: The tool was a digital health intervention implemented for use among forensic nurse examiners, law enforcement, victim support organizations, and more within four counties of California. We conducted a mixed-methods pilot study that captured data around the adoption, use, and impact of having access to the newly implemented tool. Results: The tool successfully went live in all four pilot counties at different time points with different proportions of use by county and form type: exams, referrals, addenda, risk assessments, and other. Participants were motivated to use the tool out of a perceived need for data handling functionalities that went beyond traditional manual (paper) means. Key functionalities included body mapping, data quality controls within validated forms, attaching addenda to already existing case reports, and the means to distribute data to external recipients. Further study and development are needed on functions to incorporate into body maps and forms, and understanding the information needs of law enforcement and victim support organizations. Conclusions: Our evaluation demonstrated the feasibility and acceptability of a health IT tool to support forensic nurse documentation of IPV and SA, and direct information to multiple legal and support-related stakeholders. Areas of future development include integrating IPV and SA-related data standards for digitized forms, enhancements to the body mapping feature, and understanding the needs of those who receive digital data from forensic nurse examiners within the tool.

BACKGROUND:Hospital settings provide a unique opportunity to screen for intimate partner violence (IPV) and sexual assault (SA) yet often lack health information technology (IT) solutions for generating reliable and valid medicolegal documentation via forensic reports. OBJECTIVES:The objective of the project was to evaluate a pilot, technology "tool" for documenting cases of IPV and SA that could support forensic nurse examiners and related stakeholders in generating high-quality documentation and coordinating victim support services. METHODS:The tool was a digital health intervention implemented for use among forensic nurse examiners, law enforcement, victim support organizations, and more within four counties of California. We conducted a mixed-methods pilot study that captured data around the adoption, use, and impact of having access to the newly implemented tool. RESULTS:The tool successfully went live in all four pilot counties at different time points with different proportions of use by county and form type: exams, referrals, addenda, risk assessments, and other. Participants were motivated to use the tool out of a perceived need for data handling functionalities that went beyond traditional manual (paper) means. Key functionalities included body mapping, data quality controls within validated forms, attaching addenda to already existing case reports, and the means to distribute data to external recipients. Further study and development are needed on functions to incorporate into body maps and forms and understanding the information needs of law enforcement and victim support organizations. CONCLUSION:Our evaluation demonstrated the feasibility and acceptability of a health IT tool to support forensic nurse documentation of IPV and SA and direct information to multiple legal and support-related stakeholders. Areas of future development include integrating IPV- and SA-related data standards for digitized forms, enhancements to the body mapping feature, and understanding the needs of those who receive digital data from forensic nurse examiners within the tool.

OBJECTIVES:To examine whether comfort with the use of ChatGPT in society differs from comfort with other uses of AI in society and to identify whether this comfort and other patient characteristics such as trust, privacy concerns, respect, and tech-savviness are associated with expected benefit of the use of ChatGPT for improving health. MATERIALS AND METHODS:We analyzed an original survey of U.S. adults using the NORC AmeriSpeak Panel (n = 1787). We conducted paired t-tests to assess differences in comfort with AI applications. We conducted weighted univariable regression and 2 weighted logistic regression models to identify predictors of expected benefit with and without accounting for trust in the health system. RESULTS:Comfort with the use of ChatGPT in society is relatively low and different from other, common uses of AI. Comfort was highly associated with expecting benefit. Other statistically significant factors in multivariable analysis (not including system trust) included feeling respected and low privacy concerns. Females, younger adults, and those with higher levels of education were less likely to expect benefits in models with and without system trust, which was positively associated with expecting benefits (P = 1.6 × 10-11). Tech-savviness was not associated with the outcome. DISCUSSION:Understanding the impact of large language models (LLMs) from the patient perspective is critical to ensuring that expectations align with performance as a form of calibrated trust that acknowledges the dynamic nature of trust. CONCLUSION:Including measures of system trust in evaluating LLMs could capture a range of issues critical for ensuring patient acceptance of this technological innovation.

Objectives:To report lessons from integrating the methods and perspectives of clinical informatics (CI) and implementation science (IS) in the context of Improving the Management of symPtoms during and following Cancer Treatment (IMPACT) Consortium pragmatic trials. Materials and Methods:IMPACT informaticists, trialists, and implementation scientists met to identify challenges and solutions by examining robust case examples from 3 Research Centers that are deploying systematic symptom assessment and management interventions via electronic health records (EHRs). Investigators discussed data collection and CI challenges, implementation strategies, and lessons learned. Results:CI implementation strategies and EHRs systems were utilized to collect and act upon symptoms and impairments in functioning via electronic patient-reported outcomes (ePRO) captured in ambulatory oncology settings. Limited EHR functionality and data collection capabilities constrained the ability to address IS questions. Collecting ePRO data required significant planning and organizational champions adept at navigating ambiguity. Discussion:Bringing together CI and IS perspectives offers critical opportunities for monitoring and managing cancer symptoms via ePROs. Discussions between CI and IS researchers identified and addressed gaps between applied informatics implementation and theory-based IS trial and evaluation methods. The use of common terminology may foster shared mental models between CI and IS communities to enhance EHR design to more effectively facilitate ePRO implementation and clinical responses. Conclusion:Implementation of ePROs in ambulatory oncology clinics benefits from common understanding of the concepts, lexicon, and incentives between CI implementers and IS researchers to facilitate and measure the results of implementation efforts.

This commentary is in many ways a follow-on to, and elaboration of, the commentary published in the July issue of this journal.1 The previous commentary introduced three characteristics that contribute to the uniqueness of learning health systems (LHSs) as an approach to health improvement. The three characteristics introduced there were: "(1) a multi-stakeholder learning community that is focused on the (targeted) problem and collaboratively executes the entire cycle; (2) embracing, at the outset, the uncertainty of how to improve against the problem by undertaking a rigorous discovery process before any implementation takes place; and (3) supporting multiple co-occurring cycles with a socio-technical infrastructure to create a learning system." This commentary focuses on the very important third characteristic, infrastructure. It examines the role of infrastructure in the overall architecture of an LHS and describes LHS infrastructure in terms of 10 interconnected socio-technical services accompanied by a brief description of each. Like the previous commentary, this one seeks to bring an increased level of focus to discussions of LHSs and move an emerging field, what is coming to be called "Learning Health System Science",2 toward a sharper conception of its core principles. Critically, LHS infrastructure must extend beyond digital technology in order to support improvement of individual and population health. The infrastructure must be socio-technical in the sense that it incorporates the roles that a wide range of people must play at different levels of social organization: as individuals, as teams, as members of organizations, and as citizens of civil society.5 Technology, alone, only establishes a potential for health improvement through an LHS. Viewing its infrastructure in terms of socio-technical services could be beneficial in several ways beyond working toward a consensus view of LHS structure and function. Most notably, such a modular approach could lead to sharing of interoperable infrastructure components and the possibility that sharing of such components might promote the more rapid adoption of LHS methods. Moreover, compatibility of LHS architectures could enable smaller scale LHSs to compose into a single system that functions at larger scale. Logical next steps to mature LHS infrastructure would include building consensus around the constituent services and developing specifications for each one. The authors wish to thank the many members of the group developing an organizational maturity model for Learning Health Systems, a joint project of AcademyHealth and the Learning Health Community, for their insightful suggestions that helped to shape the ideas presented in this manuscript. They also wish to thank the staff of the Agency for Healthcare Research and Quality for their reviews and most helpful comments. The authors have no conflicts of interest to declare.

Background Evidence-based medicine (EBM) has the potential to improve health outcomes, but EBM has not been widely integrated into the systems used for research or clinical decision-making. There has not been a scalable and reusable computer-readable standard for distributing research results and synthesized evidence among creators, implementers, and the ultimate users of that evidence. Evidence that is more rapidly updated, synthesized, disseminated, and implemented would improve both the delivery of EBM and evidence-based health care policy. Objective This study aimed to introduce the EBM on Fast Healthcare Interoperability Resources (FHIR) project (EBMonFHIR), which is extending the methods and infrastructure of Health Level Seven (HL7) FHIR to provide an interoperability standard for the electronic exchange of health-related scientific knowledge. Methods As an ongoing process, the project creates and refines FHIR resources to represent evidence from clinical studies and syntheses of those studies and develops tools to assist with the creation and visualization of FHIR resources. Results The EBMonFHIR project created FHIR resources (ie, ArtifactAssessment, Citation, Evidence, EvidenceReport, and EvidenceVariable) for representing evidence. The COVID-19 Knowledge Accelerator (COKA) project, now Health Evidence Knowledge Accelerator (HEvKA), took this work further and created FHIR resources that express EvidenceReport, Citation, and ArtifactAssessment concepts. The group is (1) continually refining FHIR resources to support the representation of EBM; (2) developing controlled terminology related to EBM (ie, study design, statistic type, statistical model, and risk of bias); and (3) developing tools to facilitate the visualization and data entry of EBM information into FHIR resources, including human-readable interfaces and JSON viewers. Conclusions EBMonFHIR resources in conjunction with other FHIR resources can support relaying EBM components in a manner that is interoperable and consumable by downstream tools and health information technology systems to support the users of evidence.

AbstractObjectivesIntroduce the CDS-Sandbox, a cloud-based virtual machine created to facilitate Clinical Decision Support (CDS) developers and implementers in the use of FHIR- and CQL-based open-source tools and technologies for building and testing CDS artifacts.Materials and MethodsThe CDS-Sandbox includes components that enable workflows for authoring and testing CDS artifacts. Two workshops at the 2020 and 2021 AMIA Annual Symposia were conducted to demonstrate the use of the open-source CDS tools.ResultsThe CDS-Sandbox successfully integrated the use of open-source CDS tools. Both workshops were well attended. Participants demonstrated use and understanding of the workshop materials and provided positive feedback after the workshops.DiscussionThe CDS-Sandbox and publicly available tutorial materials facilitated an understanding of the leading-edge open-source CDS infrastructure components.ConclusionThe CDS-Sandbox supports integrated use of the key CDS open-source tools that may be used to introduce CDS concepts and practice to the clinical informatics community.

Background High blood pressure (HBP) affects nearly half of adults in the United States and is a major factor in heart attacks, strokes, kidney disease, and other morbidities. To reduce risk, guidelines for HBP contain more than 70 recommendations, including many related to patient behaviors, such as home monitoring and lifestyle changes. Thus, the patient’s role in controlling HBP is crucial. Patient-facing clinical decision support (CDS) tools may help patients adhere to evidence-based care, but customization is required. Objective Our objective was to understand how to adapt CDS to best engage patients in controlling HBP. Methods We conducted a mixed methods study with two phases: (1) survey-guided interviews with a limited cohort and (2) a nationwide web-based survey. Participation in each phase was limited to adults aged between 18 and 85 years who had been diagnosed with hypertension. The survey included general questions that assessed goal setting, treatment priorities, medication load, comorbid conditions, satisfaction with blood pressure (BP) management, and attitudes toward CDS, and also a series of questions regarding A/B preferences using paired information displays to assess perceived trustworthiness of potential CDS user interface options. Results We conducted 17 survey-guided interviews to gather patient needs from CDS, then analyzed results and created a second survey of 519 adults with clinically diagnosed HBP. A large majority of participants reported that BP control was a high priority (83%), had monitored BP at home (82%), and felt comfortable using technology (88%). Survey respondents found displays with more detailed recommendations more trustworthy (56%-77% of them preferred simpler displays), especially when incorporating social trust and priorities from providers and patients like them, but had no differences in action taken. Conclusions Respondents to the survey felt that CDS capabilities could help them with HBP control. The more detailed design options for BP display and recommendations messaging were considered the most trustworthy yet did not differentiate perceived actions.

Abstract Background Systematic approaches are needed to accurately characterize the dynamic use of implementation strategies and how they change over time. We describe the development and preliminary evaluation of the Longitudinal Implementation Strategy Tracking System (LISTS), a novel methodology to document and characterize implementation strategies use over time. Methods The development and initial evaluation of the LISTS method was conducted within the Improving the Management of SymPtoms during And following Cancer Treatment (IMPACT) Research Consortium (supported by funding provided through the NCI Cancer MoonshotSM). The IMPACT Consortium includes a coordinating center and three hybrid effectiveness-implementation studies testing routine symptom surveillance and integration of symptom management interventions in ambulatory oncology care settings. LISTS was created to increase the precision and reliability of dynamic changes in implementation strategy use over time. It includes three components: (1) a strategy assessment, (2) a data capture platform, and (3) a User’s Guide. An iterative process between implementation researchers and practitioners was used to develop, pilot test, and refine the LISTS method prior to evaluating its use in three stepped-wedge trials within the IMPACT Consortium. The LISTS method was used with research and practice teams for approximately 12 months and subsequently we evaluated its feasibility, acceptability, and usability using established instruments and novel questions developed specifically for this study. Results Initial evaluation of LISTS indicates that it is a feasible and acceptable method, with content validity, for characterizing and tracking the use of implementation strategies over time. Users of LISTS highlighted several opportunities for improving the method for use in future and more diverse implementation studies. Conclusions The LISTS method was developed collaboratively between researchers and practitioners to fill a research gap in systematically tracking implementation strategy use and modifications in research studies and other implementation efforts. Preliminary feedback from LISTS users indicate it is feasible and usable. Potential future developments include additional features, fewer data elements, and interoperability with alternative data entry platforms. LISTS offers a systematic method that encourages the use of common data elements to support data analysis across sites and synthesis across studies. Future research is needed to further adapt, refine, and evaluate the LISTS method in studies with employ diverse study designs and address varying delivery settings, health conditions, and intervention types.

Background Electronic clinical quality measures (eCQMs) from electronic health records (EHRs) are a key component of quality improvement (QI) initiatives in small-to-medium size primary care practices, but using eCQMs for QI can be challenging. Organizational strategies are needed to effectively operationalize eCQMs for QI in these practice settings. Objective This study aimed to characterize strategies that seven regional cooperatives participating in the EvidenceNOW initiative developed to generate and report EHR-based eCQMs for QI in small-to-medium size practices. Methods A qualitative study comprised of 17 interviews with representatives from all seven EvidenceNOW cooperatives was conducted. Interviewees included administrators were with both strategic and cooperative-level operational responsibilities and external practice facilitators were with hands-on experience helping practices use EHRs and eCQMs. A subteam conducted 1-hour semistructured telephone interviews with administrators and practice facilitators, then analyzed interview transcripts using immersion crystallization. The analysis and a conceptual model were vetted and approved by the larger group of coauthors. Results Cooperative strategies consisted of efforts in four key domains. First, cooperative adaptation shaped overall strategies for calculating eCQMs whether using EHRs, a centralized source, or a “hybrid strategy” of the two. Second, the eCQM generation described how EHR data were extracted, validated, and reported for calculating eCQMs. Third, practice facilitation characterized how facilitators with backgrounds in health information technology (IT) delivered services and solutions for data capture and quality and practice support. Fourth, performance reporting strategies and tools informed QI efforts and how cooperatives could alter their approaches to eCQMs. Conclusion Cooperatives ultimately generated and reported eCQMs using hybrid strategies because they determined neither EHRs alone nor centralized sources alone could operationalize eCQMs for QI. This required cooperatives to devise solutions and utilize resources that often are unavailable to typical small-to-medium-sized practices. The experiences from EvidenceNOW cooperatives provide insights into how organizations can plan for challenges and operationalize EHR-based eCQMs.

Introduction:While data repositories are well-established in clinical and research enterprises, knowledge repositories with shareable computable biomedical knowledge (CBK) are relatively new entities to the digital health ecosystem. Trustworthy knowledge repositories are necessary for learning health systems, but the policies, standards, and practices to promote trustworthy CBK artifacts and methods to share, and safely and effectively use them are not well studied. Methods:We conducted an online survey of 24 organizations in the United States known to be involved in the development or deployment of CBK. The aim of the survey was to assess the current policies and practices governing these repositories and to identify best practices. Descriptive statistics methods were applied to data from 13 responding organizations, to identify common practices and policies instantiating the TRUST principles of Transparency, Responsibility, User Focus, Sustainability, and Technology. Results:All 13 respondents indicated to different degrees adherence to policies that convey TRUST. Transparency is conveyed by having policies pertaining to provenance, credentialed contributors, and provision of metadata. Repositories provide knowledge in machine-readable formats, include implementation guidelines, and adhere to standards to convey Responsibility. Repositories report having Technology functions that enable end-users to verify, search, and filter for knowledge products. Less common TRUST practices are User Focused procedures that enable consumers to know about user licensing requirements or query the use of knowledge artifacts. Related to Sustainability, less than a majority post describe their sustainability plans. Few organizations publicly describe whether patients play any role in their decision-making. Conclusion:It is essential that knowledge repositories identify and apply a baseline set of criteria to lay a robust foundation for their trustworthiness leading to optimum uptake, and safe, reliable, and effective use to promote sharing of CBK. Identifying current practices suggests a set of desiderata for the CBK ecosystem in its continued evolution.

OBJECTIVE:This study examines guideline-based high blood pressure (HBP) and hypertension recommendations and evaluates the suitability and adequacy of the data and logic required for a Fast Healthcare Interoperable Resources (FHIR)-based, patient-facing clinical decision support (CDS) HBP application. HBP is a major predictor of adverse health events, including stroke, myocardial infarction, and kidney disease. Multiple guidelines recommend interventions to lower blood pressure, but implementation requires patient-centered approaches, including patient-facing CDS tools. METHODS:We defined concept sets needed to measure adherence to 71 recommendations drawn from eight HBP guidelines. We measured data quality for these concepts for two cohorts (HBP screening and HBP diagnosed) from electronic health record (EHR) data, including four use cases (screening, nonpharmacologic interventions, pharmacologic interventions, and adverse events) for CDS. RESULTS:We identified 102,443 people with diagnosed and 58,990 with undiagnosed HBP. We found that 21/35 (60%) of required concept sets were unused or inaccurate, with only 259 (25.3%) of 1,101 codes used. Use cases showed high inclusion (0.9-11.2%), low exclusion (0-0.1%), and missing patient-specific context (up to 65.6%), leading to data in 2/4 use cases being insufficient for accurate alerting. DISCUSSION:Data quality from the EHR required to implement recommendations for HBP is highly inconsistent, reflecting a fragmented health care system and incomplete implementation of standard terminologies and workflows. Although imperfect, data were deemed adequate for two test use cases. CONCLUSION:Current data quality allows for further development of patient-facing FHIR HBP tools, but extensive validation and testing is required to assure precision and avoid unintended consequences.

As part of its Cancer Moonshot, in 2018 the National Cancer Institute established an initiative to fund a consortium that aims to improve the monitoring and management of patients’ cancer-related symptoms using informatics solutions. The consortium, Improving the Management of symPtoms during And following Cancer Treatment (IMPACT), is comprised of three research centers and a coordinating center that are tasked with collecting and sharing symptom data from across the cancer care continuum – at the point-of-care in oncology clinics and via remote settings – and evaluating the effects that cancer-related symptom management tools and data have on patients, through interventions conducted in care delivery organizations. The research centers leverage informatics strategies such as novel electronic health record user interfaces, patient-directed mobile health interventions, and symptombased clinical decision support tools. This panel will discuss their efforts with developing and implementing symptom management tools, and highlight the solutions and challenges encountered. Learning objectives include understanding current issues with developing and implementing systems for tracking cancer-related symptoms and using decision support tools for effective supportive care and symptom management.

Objectives Evidence-based practice requires the use of research results to inform care. Computers can add capacity for evidence-based practice by making the information from research results, appraisals, and summarizations searchable and re-usable without labor-intensive manual screening and repetition of data entry. Such interoperability can be achieved by establishing universal standards for data exchange for communicating evidence concepts in machine-interpretable formats. Method Health Level 7 (HL7) is a standards development organization that has developed a standard for electronic exchange of healthcare information called Fast Healthcare Interoperability Resources (FHIR). We are using the HL7 standards development methodology to extend FHIR to create an Evidence Resource for exchanging descriptive, statistical and certainty concepts related to evidence. Results The FHIR Resources for Evidence-Based Medicine Knowledge Assets (EBMonFHIR) project is in active development with a substantial coalition of international organizations and coordination with other standards development groups. The Statistic Resource currently supports explicit descriptions of the populations and subgroups (exposureBackground elements), interventions or exposures and comparators (exposureVariant elements), the outcomes (measuredVariable elements), and for each statistic the sample size, the value with unit of measure, the precision estimate, the p value, and the certainty of the statistic. The project website (http://wiki.hl7.org/index.php?title=EBMonFHIR) includes multiple examples and information on how to participate. Conclusions Working together we can achieve interoperability for evidence in the electronic era to realize the technological breakthroughs we see in other domains such as navigation support. A common information architecture will also facilitate the harmonization of ‘Real World Evidence’ and ‘Evidence Based Medicine’ which collectively represent clear understanding of evidence and its certainty, regardless of evidence source. Extending the solutions achieving interoperability for healthcare services provide a means to not only solve this challenge for the Evidence Ecosystem but also to keep it well connected with healthcare services delivery.