BACKGROUND:Engagement of clinicians who understand clinical workflows and technology constraints can accelerate the development and implementation of better electronic health record (EHR) designs that improve quality and reduce burnout. Provider builder programs can accelerate clinical informatics education for a broader coalition of clinical specialties. OBJECTIVES:In this State of the Art/Best Practice paper, we aim to (1) propose a provider builder maturity model informed by the experience of three institutions using a single EHR vendor (Epic Systems) and (2) describe the program elements and relationships necessary to advance along this model to yield organizational benefits. METHODS:We used a modified version of the Glaser State-of-the-Art approach, gathering consensus among a small group of experts at institutions with successful provider builder programs. The model was updated through meetings with a larger group of experts and then feedback from presentation at national conferences and the American Medical Informatics Association's Maturity Model Working Group. RESULTS:The final maturity model describes the characteristics and suggested next steps beginning from Planting the Seed (Stage 0) and progressing through Lone Wolves (Stage 1), a Community of Builders (Stage 2), Organizational Structure (Stage 3), a Council of Builders (Stage 4), and Informatics in the Room Where it Happens (Stage 5). We also describe the journeys of three organizations through these stages. CONCLUSION:A provider builder maturity model can help guide organizations on their journey engaging clinicians in collaborative EHR design to promote quality and safety and reduce burnout.

This study aimed to evaluate the effect of optimizing the ambulatory medication preference list on provider efficiency in medication ordering.Using electronic health record (EHR) vendor data, a multidisciplinary informatics team optimized the general ambulatory medication preference list to better align with providers' ordering patterns. We conducted a pre-postintervention analysis assessing time-in-orders per encounter and number of manual changes per order.Postintervention, average manual changes per order decreased from 4.12 to 3.00 (p < 0.01), and median time spent in the orders activity per encounter decreased from 3.1 to 2.3 minutes (p < 0.01).Optimizing the ambulatory medication preference list reduced time spent and clicks needed by providers when ordering medications. This is relevant to ongoing efforts to address EHR-related burden.

Enhancing the efficiency of family-centered rounds (FCRs) while ensuring timely patient care has been a focus of study over the past decade. We employed an Operations Research technique (i.e., simulation) to identify opportunities for improving rounding efficiency on our inpatient cardiology unit at Nationwide Children's Hospital (NCH).Through simulation of schedule-based rounds, our aims were to reduce the length of stay (LOS) and subsequent healthcare costs via (1) prioritizing rounds for patients needing time-sensitive care decisions or those likely ready to be discharged, and (2) enhancing participation from both families and bedside nurses during rounds.Data were collected through direct observation of rounding activities. We then conducted simulations to evaluate the effect of various rounding paths on efficiency, measured in terms of time and penalties depending on context.Our simulations indicated a tradeoff between minimizing the risk of delayed rounding and the amount of time spent on rounds. Optimizing rounds for 20 patients reduced cumulative patient waiting time and associated penalty scores. Based on prior research linking earlier clinical interventions to improved efficiency, this approach is estimated to reduce LOS by 166.08 hours and cost by approximately $3,460 per rotation.By simulating the hospital rounding processes on an inpatient pediatric cardiology unit, we demonstrated that prioritized rounding could reduce both LOS and associated costs. Despite a potential increase in total rounding time, which can be managed by clinical decision-makers, we recommend utilizing scheduling-based FCRs based on prioritization techniques that enhance rounding efficiency while minimizing risk and cost.

OBJECTIVES:The Emergency Department Work Index (EDWIN) is a validated overcrowding score shown to correlate well with staff assessment of adult emergency department (ED) overcrowding and the potential need for diversion. It derives from the number of staffed ED beds, attending physicians on duty, patients within each triage category, and admitted patients. To date, no study has validated EDWIN in a pediatric community ED setting. We aim to determine if EDWIN correlates with established overcrowding measures and provider perception of overcrowding within a freestanding, community-based pediatric ED. METHODS:In this prospective observational study at a freestanding, community-based pediatric ED, EDWIN was calculated hourly over 8 weeks throughout the year. EDWIN was compared with other objective and previously established ED metrics of overcrowding, including rates of patients who left without being seen (LWBS), average time from arrival to ED room, average length of stay (LOS), ED occupancy rates, and number of patients in the waiting room. Furthermore, EDWIN was compared with provider perception of overcrowding by surveying providers 6 times a day during the study period using novel, real-time, longitudinal, electronic health record-based survey distribution methodology. Spearman correlation coefficients were calculated to characterize the associations between EDWIN vs provider perception and EDWIN vs ED metrics. ANOVA and Tukey HSD were used to compare means of ED metrics of overcrowding across EDWIN severity categories. RESULTS:Five hundred eleven provider perception survey responses were collected from July 2022 through January 2023. EDWIN directly correlated with all measures of overcrowding, including provider perception of crowdedness (rho = 0.67), LWBS rates (rho = 0.44), average time from arrival to ED room (rho = 0.74), average LOS (rho = 0.70), ED occupancy rates (rho = 0.68), and number of patients in the waiting room (rho = 0.65). All findings were statistically significant ( P < 0.05). CONCLUSIONS:Our findings suggest that EDWIN is an accurate tool to measure overcrowding in a freestanding, community-based pediatric ED.

When computers and other technological advances entered the world of medicine and patient care, physicians marveled at the potential and immediately grasped the benefits of information that was instantly accessible.Today, we scroll through hundreds of MRI images in seconds, transmit prescriptions digitally, and review patients' medical records on our smartphones and yet, compared to our expectations, digital innovations in medicine have still fallen short.The inefficiencies of the current electronic health record systems (EHRs) and the burden that resulted from implementing them are different than the advances that were envisioned and promised.In this article, we will describe the field of clinical informatics-a subspecialty that aims to address these gaps-discuss some of the historical context of neonatal informatics and present some recommendations to improve current documentation and EHR workflows within neonatology.

This study evaluates whether implementation of a content expert–developed clinical documentation tool can be beneficial to workflow by reducing time from patient arrival to encounter closure among pediatric patients receiving intestinal rehabilitation.

OBJECTIVE:The national developmental-behavioral pediatric (DBP) workforce struggles to meet current service demands because of several factors. Lengthy and inefficient documentation processes are likely to contribute to service demand challenges, but DBP documentation patterns have not been sufficiently studied. Identifying clinical practice patterns may inform strategies to address documentation burden in DBP practice.METHODS:Nearly 500 DBP physicians in the United States use a single commercial electronic health record (EHR) system (EpicCare Ambulatory, Epic Systems Corporation, Verona WI). We evaluated descriptive statistics using the US Epic DBP provider data set. We then compared DBP documentation metrics against those of pediatric primary care and selected pediatric subspecialty providers who provide similar types of care. One-way analyses of variance (ANOVAs) were conducted to determine whether outcomes differed among provider specialties.RESULTS:We identified 4 groups for analysis from November 2019 through February 2020: DBP (n = 483), primary care (n = 76,423), pediatric psychiatry (n = 783), and child neurology (n = 8589). Post hoc pairwise comparisons revealed statistically significant differences between multiple outcome-specialty combinations. Time in notes per appointment and progress note length demonstrated the strongest evidence of an increased burden on DBP providers compared with other similar provider groups.CONCLUSION:DBP providers spend a significant amount of time documenting progress notes both during and outside of normal clinic hours. This preliminary analysis highlights the utility of using EHR user activity data to quantitatively measure documentation burden.

OBJECTIVES:Develop and deploy a disease cohort-based machine learning algorithm for timely identification of hospitalized pediatric patients at risk for clinical deterioration that outperforms our existing situational awareness program. DESIGN:Retrospective cohort study. SETTING:Nationwide Children's Hospital, a freestanding, quaternary-care, academic children's hospital in Columbus, OH. PATIENTS:All patients admitted to inpatient units participating in the preexisting situational awareness program from October 20, 2015, to December 31, 2019, excluding patients over 18 years old at admission and those with a neonatal ICU stay during their hospitalization. INTERVENTIONS:We developed separate algorithms for cardiac, malignancy, and general cohorts via lasso-regularized logistic regression. Candidate model predictors included vital signs, supplemental oxygen, nursing assessments, early warning scores, diagnoses, lab results, and situational awareness criteria. Model performance was characterized in clinical terms and compared with our previous situational awareness program based on a novel retrospective validation approach. Simulations with frontline staff, prior to clinical implementation, informed user experience and refined interdisciplinary workflows. Model implementation was piloted on cardiology and hospital medicine units in early 2021. MEASUREMENTS AND MAIN RESULTS:The Deterioration Risk Index (DRI) was 2.4 times as sensitive as our existing situational awareness program (sensitivities of 53% and 22%, respectively; p < 0.001) and required 2.3 times fewer alarms per detected event (121 DRI alarms per detected event vs 276 for existing program). Notable improvements were a four-fold sensitivity gain for the cardiac diagnostic cohort (73% vs 18%; p < 0.001) and a three-fold gain (81% vs 27%; p < 0.001) for the malignancy diagnostic cohort. Postimplementation pilot results over 18 months revealed a 77% reduction in deterioration events (three events observed vs 13.1 expected, p = 0.001). CONCLUSIONS:The etiology of pediatric inpatient deterioration requires acknowledgement of the unique pathophysiology among cardiology and oncology patients. Selection and weighting of diverse candidate risk factors via machine learning can produce a more sensitive early warning system for clinical deterioration. Leveraging preexisting situational awareness platforms and accounting for operational impacts of model implementation are key aspects to successful bedside translation.

Background Many of the benefits of electronic health records (EHRs) have not been achieved at expected levels because of a variety of unintended negative consequences such as documentation burden. Previous studies have characterized EHR use during and outside work hours, with many reporting that physicians spend considerable time on documentation-related tasks. These studies characterized EHR use during and outside work hours using clock time versus actual physician clinic schedules to define the outside work time. Objective This study aimed to characterize EHR work outside scheduled clinic hours among primary care pediatricians using a retrospective descriptive task analysis of EHR access log data and actual physician clinic schedules to define work time. Methods We conducted a retrospective, exploratory, descriptive task analysis of EHR access log data from primary care pediatricians in September 2019 at a large Midwestern pediatric health center to quantify and identify actions completed outside scheduled clinic hours. Mixed-effects statistical modeling was used to investigate the effects of age, sex, clinical full-time equivalent status, and EHR work during scheduled clinic hours on the use of EHRs outside scheduled clinic hours. Results Primary care pediatricians (n=56) in this study generated 1,523,872 access log data points (across 1069 physician workdays) and spent an average of 4.4 (SD 2.0) hours and 0.8 (SD 0.8) hours per physician per workday engaged in EHRs during and outside scheduled clinic hours, respectively. Approximately three-quarters of the time working in EHR during or outside scheduled clinic hours was spent reviewing data and reports. Mixed-effects regression revealed no associations of age, sex, or clinical full-time equivalent status with EHR use during or outside scheduled clinic hours. Conclusions For every hour primary care pediatricians spent engaged with the EHR during scheduled clinic hours, they spent approximately 10 minutes interacting with the EHR outside scheduled clinic hours. Most of their time (during and outside scheduled clinic hours) was spent reviewing data, records, and other information in EHR.

BACKGROUND:With the increased sharing of electronic health information as required by the US 21st Century Cures Act, there is an increased risk of breaching patient, parent, or guardian confidentiality. The prevalence of sensitive terms in clinical notes is not known. OBJECTIVE:The aim of this study is to define sensitive terms that represent the documentation of content that may be private and determine the prevalence and characteristics of provider notes that contain sensitive terms. METHODS:Using keyword expansion, we defined a list of 781 sensitive terms. We searched all provider history and physical, progress, consult, and discharge summary notes for patients aged 0-21 years written between January 1, 2019, and December 31, 2019, for a direct string match of sensitive terms. We calculated the prevalence of notes with sensitive terms and characterized clinical encounters and patient characteristics. RESULTS:Sensitive terms were present in notes from every clinical context in all pediatric ages. Terms related to the mental health category were most used overall (254,975/1,338,297, 19.5%), but terms related to substance abuse and reproductive health were most common in patients aged 0-3 years. History and physical notes (19,854/34,771, 57.1%) and ambulatory progress notes (265,302/563,273, 47.1%) were most likely to include sensitive terms. The highest prevalence of notes with sensitive terms was found in pain management (950/1112, 85.4%) and child abuse (1092/1282, 85.2%) clinics. CONCLUSIONS:Notes containing sensitive terms are not limited to adolescent patients, specific note types, or certain specialties. Recognition of sensitive terms across all ages and clinical settings complicates efforts to protect patient and caregiver privacy in the era of information-blocking regulations.

Abstract Objective A large amount of clinical data are stored in clinical notes that frequently contain spelling variations, typos, local practice-generated acronyms, synonyms, and informal words. Instead of relying on established but infrequently updated ontologies with keywords limited to formal language, we developed an artificial intelligence (AI) assistant (named “DeepSuggest”) that interactively offers suggestions to expand or pivot queries to help overcome these challenges. Methods We applied an unsupervised neural network (Word2Vec) to the clinical notes to build keyword contextual similarity matrix. With a user's input query, DeepSuggest generates a list of relevant keywords, including word variations (e.g., formal or informal forms, synonyms, abbreviations, and misspellings) and other relevant words (e.g., related diagnosis, medications, and procedures). Human intelligence is then used to further refine or pivot their query. Results DeepSuggest learns the semantic and linguistic relationships between the words from a large collection of local notes. Although DeepSuggest is only able to recall 0.54 of Systematized Nomenclature of Medicine (SNOMED) synonyms on average among the top 60 suggested terms, it covers the semantic relationship in our corpus for a larger number of raw concepts (6.3 million) than SNOMED ontology (24,921) and is able to retrieve terms that are not stored in existing ontologies. The precision for the top 60 suggested words averages at 0.72. Usability test resulted that DeepSuggest is able to achieve almost twice the recall on clinical notes compared with Epic (average of 5.6 notes retrieved by DeepSuggest compared with 2.6 by Epic). Conclusion DeepSuggest showed the ability to improve retrieval of relevant clinical notes when implemented on a local corpus by suggesting spelling variations, acronyms, and semantically related words. It is a promising tool in helping users to achieve a higher recall rate for clinical note searches and thus boosting productivity in clinical practice and research. DeepSuggest can supplement established ontologies for query expansion.

BACKGROUND Many of the benefits of electronic health records (EHRs) have not been achieved at expected levels due to a variety of unintended negative consequences such as documentation burden. Previous studies have characterized EHR use during and outside work hours, with many reporting physicians spending considerable time on documentation-related tasks. These studies characterized EHR use during and outside work hours using clock time versus actual physician schedules to define outside work time. OBJECTIVE This study closes a knowledge gap by characterizing EHR engagement outside work hours using actual physician schedules to define EHR work outside work hours. METHODS A retrospective exploratory descriptive task analysis of EHR access log data from primary care pediatricians in September 2019 at a large midwestern pediatric health center was conducted to quantify and identify actions completed outside work hours. Mixed effects statistical modeling was used to investigate the effects of age, sex, clinical full-time equivalent status, and EHR engagement during work hours on the use of EHRs outside work hours. RESULTS Primary care pediatricians (n=56) in this study generated a total of 1,523,872 access-log data points (across 1,069 physician workdays) and spent an average of 3.9 and 1.2 hours per physician per workday engaged in the EHR during and outside work hours, respectively. About three-quarters of the time engaged in the EHR during or outside work hours was spent reviewing data and reports. Mixed effects regression revealed no associations of age, sex, nor clinical full-time equivalent status with EHR use during or outside work hours. CONCLUSIONS For every hour primary care pediatricians in this study spent engaged with the EHR during work hours, they spent about 20 minutes interacting with the EHR outside work hours. Most of their time (both during and outside of work hours) was spent reviewing data, records, and other information in the EHR.

IMPORTANCE Patient portals can be configured to allow confidential communication for adolescents' sensitive health care information. Guardian access of adolescent patient portal accounts could compromise adolescents' confidentiality. OBJECTIVE To estimate the prevalence of guardian access to adolescent patient portals at 3 academic children's hospitals. DESIGN, SETTING, AND PARTICIPANTS A cross-sectional study to estimate the prevalence of guardian access to adolescent patient portal accounts was conducted at 3 academic children's hospitals. Adolescent patients (aged 13-18 years) with access to their patient portal account with at least 1 outbound message from their portal during the study period were included. A rule-based natural language processing algorithm was used to analyze all portal messages from June 1, 2014, to February 28, 2020, and identify any message sent by guardians. The sensitivity and specificity of the algorithm at each institution was estimated through manual review of a stratified subsample of patient accounts. The overall proportion of accounts with guardian access was estimated after correcting for the sensitivity and specificity of the natural language processing algorithm. EXPOSURES Use of patient portal. MAIN OUTCOME AND MEASURES Percentage of adolescent portal accounts indicating guardian access. RESULTS A total of 3429 eligible adolescent accounts containing 25 642 messages across 3 institutions were analyzed. A total of 1797 adolescents (52%) were female and mean (SD) age was 15.6 (1.6) years. The percentage of adolescent portal accounts with apparent guardian access ranged from 52% to 57% across the 3 institutions. After correcting for the sensitivity and specificity of the algorithm based on manual review of 200 accounts per institution, an estimated 64% (95% CI, 59%-69%) to 76% (95% CI, 73%-88%) of accounts with outbound messages were accessed by guardians across the 3 institutions. CONCLUSIONS AND RELEVANCE In this study, more than half of adolescent accounts with outbound messages were estimated to have been accessed by guardians at least once. These findings have implications for health systems intending to rely on separate adolescent accounts to protect adolescent confidentiality.

Introduction: Pediatric in-hospital cardiac arrests and emergent transfers to the pediatric intensive care unit (ICU) represent a serious patient safety concern with associated increased morbidity and mortality. Some institutions have turned to the electronic health record and predictive analytics in search of earlier and more accurate detection of patients at risk for decompensation. Methods: Objective electronic health record data from 2011 to 2017 was utilized to develop an automated early warning system score aimed at identifying hospitalized children at risk of clinical deterioration. Five vital sign measurements and supplemental oxygen requirement data were used to build the Vitals Risk Index (VRI) model, using multivariate logistic regression. We compared the VRI to the hospital’s existing early warning system, an adaptation of Monaghan’s Pediatric Early Warning Score system (PEWS). The patient population included hospitalized children 18 years of age and younger while being cared for outside of the ICU. This dataset included 158 case hospitalizations (102 emergent transfers to the ICU and 56 “code blue” events) and 135,597 control hospitalizations. Results: When identifying deteriorating patients 2 hours before an event, there was no significant difference between Pediatric Early Warning Score and VRI’s areas under the receiver operating characteristic curve at false-positive rates ≤ 10% (pAUC10 of 0.065 and 0.064, respectively; P = 0.74), a threshold chosen to compare the 2 approaches under clinically tolerable false-positive rates. Conclusions: The VRI represents an objective, simple, and automated predictive analytics tool for identifying hospitalized pediatric patients at risk of deteriorating outside of the ICU setting.

OBJECTIVE:To develop a diagnostic error index (DEI) aimed at providing a practical method to identify and measure serious diagnostic errors. STUDY DESIGN:A quality improvement (QI) study at a quaternary pediatric medical center. Five well-defined domains identified cases of potential diagnostic errors. Identified cases underwent an adjudication process by a multidisciplinary QI team to determine if a diagnostic error occurred. Confirmed diagnostic errors were then aggregated on the DEI. The primary outcome measure was the number of monthly diagnostic errors. RESULTS:From January 2017 through June 2019, 105 cases of diagnostic error were identified. Morbidity and mortality conferences, institutional root cause analyses, and an abdominal pain trigger tool were the most frequent domains for detecting diagnostic errors. Appendicitis, fractures, and nonaccidental trauma were the 3 most common diagnoses that were missed or had delayed identification. CONCLUSIONS:A QI initiative successfully created a pragmatic approach to identify and measure diagnostic errors by utilizing a DEI. The DEI established a framework to help guide future initiatives to reduce diagnostic errors.

Background Increased adoption of electronic health records (EHR) with integrated clinical decision support (CDS) systems has reduced some sources of error but has led to unintended consequences including alert fatigue. The "pop-up" or interruptive alert is often employed as it requires providers to acknowledge receipt of an alert by taking an action despite the potential negative effects of workflow interruption. We noted a persistent upward trend of interruptive alerts at our institution and increasing requests for new interruptive alerts. Objectives Using Institute for Healthcare Improvement (IHI) quality improvement (QI) methodology, the primary objective was to reduce the total volume of interruptive alerts received by providers. Methods We created an interactive dashboard for baseline alert data and to monitor frequency and outcomes of alerts as well as to prioritize interventions. A key driver diagram was developed with a specific aim to decrease the number of interruptive alerts from a baseline of 7,250 to 4,700 per week (35%) over 6 months. Interventions focused on the following key drivers: appropriate alert display within workflow, clear alert content, alert governance and standardization, user feedback regarding overrides, and respect for user knowledge. Results A total of 25 unique alerts accounted for 90% of the total interruptive alert volume. By focusing on these 25 alerts, we reduced interruptive alerts from 7,250 to 4,400 per week. Conclusion Systematic and structured improvements to interruptive alerts can lead to overall reduced interruptive alert burden. Using QI methods to prioritize our interventions allowed us to maximize our impact. Further evaluation should be done on the effects of reduced interruptive alerts on patient care outcomes, usability heuristics on cognitive burden, and direct feedback mechanisms on alert utility.

Background Communication and comprehension of medical information are known barriers in health communication and equity, especially for non–English-speaking caregivers of children with special health care needs. Objective The objective of this proposal was to develop an interoperable and scalable medical translation app for non–English-speaking caregivers to facilitate the conversation between provider and caregiver/patient. Methods We employed user-centered and participatory design methods to understand the problems and develop a solution by engaging the stakeholder team (including caregivers, physicians, researchers, clinical informaticists, nurses, developers, nutritionists, pharmacists, and interpreters) and non–English-speaking caregiver participants. Results Considering the lack of interpreter service accessibility and advancement in translation technology, our team will develop and test an integrated, multimodal (voice-interactive and text-based) patient portal communication and translation app to enable non–English-speaking caregivers to communicate with providers using their preferred languages. For this initial prototype, we will focus on the Spanish language and Spanish-speaking families to test technical feasibility and evaluate usability. Conclusions Our proposal brings a unique perspective to medical translation and communication between caregiver and provider by (1) enabling voice entry and transcription in health care communications, (2) integrating with patient portals to facilitate caregiver and provider communications, and (3) adopting a translation verification model to improve accuracy of artificial intelligence–facilitated translations. Expected outcomes include improved health communications, literacy, and health equity. In addition, data points will be collected to improve autotranslation services in medical communications. We believe our proposed solution is affordable, interoperable, and scalable for health systems.

Purpose Unrecognized clinical deterioration outside the ICU is a serious patient safety concern, with only 43% of pediatric patients surviving to discharge following an in-hospital cardiac arrest Girotra et al., 2013). Past efforts to reduce the impact of deterioration events among non-ICU patients have focused on earlier detection of deterioration via early warning systems (EWS) and faster response times for imminent events. While rapid response teams have improved inpatient mortality rates, existing EWS in this domain have been underwhelming. Further, a chasm remains between development and evaluation of machine learning algorithms in silico (on computers) and the application of these algorithms in clinical settings. To improve …