Large language models (LLMs) show promise in drafting responses to patient portal messages, yet their integration into clinical workflows raises various concerns, including whether they would actually save clinicians time and effort in their portal workload. We investigate LLM alignment with individual clinicians through a comprehensive evaluation of the patient message response drafting task. We develop a novel taxonomy of thematic elements in clinician responses and propose a novel evaluation framework for assessing clinician editing load of LLM-drafted responses at both content and theme levels. We release an expert-annotated dataset and conduct large-scale evaluations of local and commercial LLMs using various adaptation techniques including thematic prompting, retrieval-augmented generation, supervised fine-tuning, and direct preference optimization. Our results reveal substantial epistemic uncertainty in aligning LLM drafts with clinician responses. While LLMs demonstrate capability in drafting certain thematic elements, they struggle with clinician-aligned generation in other themes, particularly question asking to elicit further information from patients. Theme-driven adaptation strategies yield improvements across most themes. Our findings underscore the necessity of adapting LLMs to individual clinician preferences to enable reliable and responsible use in patient-clinician communication workflows.

BACKGROUND:Many individuals with abnormal cervical cancer screening test results do not receive timely follow-up care. Clinical decision support systems (CDSS) to improve follow-up are challenged by difficulty identifying clinical elements and applying complex guideline recommendations. As part of a multisite trial, two CDSS models were implemented: one used natural language processes to evaluate extracted data outside of the electronic health record (EHR) (System A); the other used commercial EHR functionality using LOINC-defined result fields (System B). This secondary analysis compared the accuracy and trial outcomes among sites using these two CDSS models. METHODS:Primary care clinics (32 in System A and 12 in System B) were randomly assigned to usual care, CDSS alone, or CDSS with patient outreach with or without navigation. CDSS identified individuals with overdue abnormal screening results and specified the recommended follow-up and time interval. CDSS accuracy was assessed by manual chart review. Patient outreach consisted of portal/mailed letters plus a single phone call. Navigation included one or more phone calls to address barriers to care. Completion of recommended follow-up at 120 days after enrollment was the primary outcome. Clinic was the unit of randomization, and the patient was the unit of analysis. RESULTS:Between October 2020 and December 2021, 2596 patients with abnormal results were identified by the CDSS. CDSS true positives were 61.3 % in System A and 70.4 % in System B. CDSS alone versus usual care did not improve outcomes in either system. CDSS with patient outreach with or without navigation versus usual care significantly increased follow-up rates in System A (38.2 % or 37.2 % vs 23.5 %, p < 0.001) and System B (25.4 % or 23 % vs. 19.7 %, p = 0.044). CONCLUSIONS:Two CDSS models developed to identify overdue abnormal cervical cancer screening test results had moderate accuracy. Both models with patient outreach with or without navigation - but not CDSS alone - increased recommended follow-up. Future CDSS for cervical cancer screening may be improved with open-source tools developed in public-private partnerships.

Follow-up question generation is an essential feature of dialogue systems as it can reduce conversational ambiguity and enhance modeling complex interactions. Conversational contexts often pose core NLP challenges such as (i) extracting relevant information buried in fragmented data sources, and (ii) modeling parallel thought processes. These two challenges occur frequently in medical dialogue as a doctor asks questions based not only on patient utterances but also their prior EHR data and current diagnostic hypotheses. Asking medical questions in asynchronous conversations compounds these issues as doctors can only rely on static EHR information to motivate follow-up questions. To address these challenges, we introduce FollowupQ, a novel framework for enhancing asynchronous medical conversation. FollowupQ is a multi-agent framework that processes patient messages and EHR data to generate personalized follow-up questions, clarifying patient-reported medical conditions. FollowupQ reduces requisite provider follow-up communications by 34%. It also improves performance by 17% and 5% on real and synthetic data, respectively. We also release the first public dataset of asynchronous medical messages with linked EHR data alongside 2,300 follow-up questions written by clinical experts for the wider NLP research community.

BACKGROUND:Suboptimal communication between clinicians remains a frequent driver of preventable adverse health care-related events, increased costs, and patient and physician dissatisfaction.METHODS:Cross-sectional surveys on preoperative interspecialty communication, tailored by stakeholder type, were administered to (1) primary care providers in northern New England, (2) anesthesia providers working in the perioperative clinic of a tertiary rural academic medical center, (3) surgeons from the same center, and (4) older surgical patients who underwent preoperative assessment at the same center.RESULTS:In total, 107/249 (43.0%) providers and 103/265 (39.9%) patients completed the survey. Preoperative communication was perceived as logistically challenging (59.8%), particularly across health systems. More than 77% of anesthesia and surgery providers indicated that they communicate frequently or sometimes, but 92.5% of primary care providers indicated that they rarely or never communicate with anesthesia providers. Some of the most common reasons for preoperative communication were discussion of complex patients, perioperative medication management, and optimization of comorbidities. Although 96.1% of older surgical patients reported that preoperative communication between providers is important, only 40.4% felt that their providers communicate very or extremely well. Many patients emphasized the importance of preoperative communication between providers to ensure transfer of critical clinical information.CONCLUSION:Surgeons and anesthesiologists infrequently communicate with primary care providers in one rural tertiary center, in contrast to patient expectations and values. These study results will help identify priorities and potentially resolvable barriers to bridging the gap between the inpatient perioperative and outpatient primary care teams. Future studies should focus on strategies to improve communication between hospital and community providers to prevent complications and readmission.

Timely follow-up after an abnormal cancer screening test result is needed to maximize the benefits of screening, but is frequently not achieved. Little is known about patient experiences with the process of following up abnormal screening results. Assess patient experiences and perceptions regarding the process of a diagnostic workup following abnormal breast, cervical, or colorectal cancer screening results. Survey of participating patients between April 2021 and June 2022 after reaching the primary outcome time point in a randomized controlled trial to improve follow-up of overdue abnormal screening results. Patients from 44 participating practices in three primary care practice networks. Self-reported ease of scheduling follow-up, perceived barriers or concerns, provider trust, and satisfaction with communication and care received for the follow-up of abnormal screening results. Overall, 241 (25.0

Importance:Realizing the benefits of cancer screening requires testing of eligible individuals and processes to ensure follow-up of abnormal results. Objective:To test interventions to improve timely follow-up of overdue abnormal breast, cervical, colorectal, and lung cancer screening results. Design, Setting, and Participants:Pragmatic, cluster randomized clinical trial conducted at 44 primary care practices within 3 health networks in the US enrolling patients with at least 1 abnormal cancer screening test result not yet followed up between August 24, 2020, and December 13, 2021. Intervention:Automated algorithms developed using data from electronic health records (EHRs) recommended follow-up actions and times for abnormal screening results. Primary care practices were randomized in a 1:1:1:1 ratio to (1) usual care, (2) EHR reminders, (3) EHR reminders and outreach (a patient letter was sent at week 2 and a phone call at week 4), or (4) EHR reminders, outreach, and navigation (a patient letter was sent at week 2 and a navigator outreach phone call at week 4). Patients, physicians, and practices were unblinded to treatment assignment. Main Outcomes and Measures:The primary outcome was completion of recommended follow-up within 120 days of study enrollment. The secondary outcomes included completion of recommended follow-up within 240 days of enrollment and completion of recommended follow-up within 120 days and 240 days for specific cancer types and levels of risk. Results:Among 11 980 patients (median age, 60 years [IQR, 52-69 years]; 64.8% were women; 83.3% were White; and 15.4% were insured through Medicaid) with an abnormal cancer screening test result for colorectal cancer (8245 patients [69%]), cervical cancer (2596 patients [22%]), breast cancer (1005 patients [8%]), or lung cancer (134 patients [1%]) and abnormal test results categorized as low risk (6082 patients [51%]), medium risk (3712 patients [31%]), or high risk (2186 patients [18%]), the adjusted proportion who completed recommended follow-up within 120 days was 31.4% in the EHR reminders, outreach, and navigation group (n = 3455), 31.0% in the EHR reminders and outreach group (n = 2569), 22.7% in the EHR reminders group (n = 3254), and 22.9% in the usual care group (n = 2702) (adjusted absolute difference for comparison of EHR reminders, outreach, and navigation group vs usual care, 8.5% [95% CI, 4.8%-12.0%], P < .001). The secondary outcomes showed similar results for completion of recommended follow-up within 240 days and by subgroups for cancer type and level of risk for the abnormal screening result. Conclusions and Relevance:A multilevel primary care intervention that included EHR reminders and patient outreach with or without patient navigation improved timely follow-up of overdue abnormal cancer screening test results for breast, cervical, colorectal, and lung cancer. Trial Registration:ClinicalTrials.gov Identifier: NCT03979495.

Individuals with Down syndrome (DS) are at an increased risk of numerous autoimmune conditions, most commonly autoimmune hypothyroidism. 1 Bull M.J. Trotter T. Santoro S.L. et al. Health supervision for children and adolescents with Down syndrome. Pediatrics. 2022; 149e2022057010https://doi.org/10.1542/peds.2022-057010 Crossref PubMed Google Scholar This association is likely to be secondary to complex immune dysregulation caused by trisomy of chromosome 21, in particular the autoimmune regulator gene and interferon dysfunction. Alopecia areata (AA) is reportedly more common in people with DS, but prevalence studies are limited, with figures varying from 1.3% to 11% compared with 0.1% to 0.2% in the general population. 2 Ryan C. Vellody K. Belazarian L. Rork J.F. Dermatologic conditions in Down syndrome. Pediatr Dermatol. 2021; 38: 49-57https://doi.org/10.1111/pde.14731 Crossref PubMed Scopus (8) Google Scholar Dermatologists often question when to evaluate thyroid function in patients with AA. Furthermore, given the increased risk of hypothyroidism in individuals with DS, the question is raised on how thyroid screening should be approached in patients with DS when AA is diagnosed. In a review of 298 pediatric patients with AA, Patel et al 3 Patel D. Li P. Bauer A.J. Castelo-Soccio L. Screening guidelines for thyroid function in children with alopecia areata. JAMA Dermatol. 2017; 153: 1307-1310https://doi.org/10.1001/jamadermatol.2017.3694 Crossref PubMed Scopus (29) Google Scholar found thyroid abnormalities were more common in patients with AA with DS (P = 0.004), and thus, recommended thyroid screening.

Importance:Health care systems focus on delivering routine cancer screening to eligible individuals, yet little is known about the perceptions of primary care practitioners (PCPs) about barriers to timely follow-up of abnormal results. Objective:To describe PCP perceptions about factors associated with the follow-up of abnormal breast, cervical, colorectal, and lung cancer screening test results. Design, Setting, and Participants:Survey study of PCPs from 3 primary care practice networks in New England between February and October 2020, prior to participating in a randomized clinical trial to improve follow-up of abnormal cancer screening test results. Participants were physicians and advanced practice clinicians from participating practices. Main Outcomes and Measures:Self-reported process, attitudes, knowledge, and satisfaction about the follow-up of abnormal cancer screening test results. Results:Overall, 275 (56.7%) PCPs completed the survey (range by site, 34.9%-71.9%) with more female PCPs (61.8% [170 of 275]) and general internists (73.1% [201 of 275]); overall, 28,7% (79 of 275) were aged 40 to 49 years. Most PCPs felt responsible for managing abnormal cancer screening test results with the specific cancer type being the best factor (range, 63.6% [175 of 275] for breast to 81.1% [223 of 275] for lung; P < .001). The PCPs reported limited support for following up on overdue abnormal cancer screening test results. Standard processes such as automated reports, reminder letters, or outreach workers were infrequently reported. Major barriers to follow-up of abnormal cancer screening test results across all cancer types included limited electronic health record tools (range, 28.5% [75 of 263]-36.5%[96 of 263]), whereas 50% of PCPs felt that there were major social barriers to receiving care for abnormal cancer screening test results for colorectal cancer. Fewer than half reported being very satisfied with the process of managing abnormal cancer screening test results, with satisfaction being greatest for breast cancer (46.9% [127 of 271]) and lowest for cervical (21.8% [59 of 271]) and lung cancer (22.4% [60 of 268]). Conclusions and Relevance:In this survey study of PCPs, important deficiencies in systems for managing abnormal cancer screening test results were reported. These findings suggest a need for comprehensive organ-agnostic systems to promote timely follow-up of abnormal cancer screening results using a primary care-focused approach across the range of cancer screening tests.

BACKGROUND:Transitional Care Management (TCM) is a reimbursable service designed to minimize hospital readmissions. We describe a multifaceted approach to increase TCM services among 107 primary care providers in a rural catchment area of 4250 square miles.OBJECTIVE:The primary objective was to increase use of TCM phone calls, office visits, and billing codes; the secondary objective was to decrease hospital readmissions.METHODS:We utilized a learning health system model, an improvement support team (IST), and a learning collaborative that included webinars and in-person support. The process emphasized user-centered system redesign, coaching, electronic health record (EHR) improvements, and real-time feedback. Analyses included statistical process control charts, box plots, analysis of variance, and t-tests.RESULTS:The IST engaged stakeholders to design and test TCM workflows and EHR prototypes. This resulted in rapid, iterative improvements and system-wide spread of new processes. In the month following implementation, TCM calls and visits quadrupled and increased during 18 subsequent months. Pragmatically, most discharged patients (95% in a subsample) did not receive both the TCM call and visit, serving as a comparison group. The Readmission rate for patients receiving complete TCM services was 5.0% (n = 101) versus 11.9% for comparators (n = 2103, P = .03). Billing codes increased initially, then returned to baseline.CONCLUSIONS:Our approach led to rapid, sustained scaling of TCM calls and visits in a rural primary care group. Patients who received TCM calls and visits had significantly fewer readmissions. Training of new staff, including PCPs, is required for sustainability. Future research is warranted to increase adoption and evaluate additional outcomes including mortality rates, patient satisfaction, and health care economics.

INTRODUCTION:While substantial attention is focused on the delivery of routine preventive cancer screening, less attention has been paid to systematically ensuring that there is timely follow-up of abnormal screening test results. Barriers to completion of timely follow-up occur at the patient, provider, care team and system levels. METHODS:In this pragmatic cluster randomized controlled trial, primary care sites in three networks are randomized to one of four arms: (1) standard care, (2) "visit-based" reminders that appear in a patient's electronic health record (EHR) when it is accessed by either patient or providers (3) visit based reminders with population health outreach, and (4) visit based reminders, population health outreach, and patient navigation with systematic screening and referral to address social barriers to care. Eligible patients in participating practices are those overdue for follow-up of an abnormal results on breast, cervical, colorectal and lung cancer screening tests. RESULTS:The primary outcome is whether an individual receives follow-up, specific to the organ type and screening abnormality, within 120 days of becoming eligible for the trial. Secondary outcomes assess the effect of intervention components on the patient and provider experience of obtaining follow-up care and the delivery of the intervention components. CONCLUSIONS:This trial will provide evidence for the role of a multilevel intervention on improving the follow-up of abnormal cancer screening test results. We will also specifically assess the relative impact of the components of the intervention, compared to standard care. TRIAL REGISTRATION:ClinicalTrials.gov NCT03979495.

BACKGROUND:Electronic health records (EHRs) are a key tool for primary care practice. However, EHR functionality is not keeping pace with the evolving informational and decision-support needs of behavioral health clinicians (BHCs) working on integrated teams.OBJECTIVE:Describe workflows and tasks of BHCs working with integrated teams, identify their health information technology needs, and develop EHR tools to address them.METHOD:A mixed-methods, comparative-case study of six community health centers (CHCs) in Oregon, each with at least one BHC integrated in their primary care team. We observed clinical work and conducted interviews to understand workflows and clinical tasks, aiming to identify how effectively current EHRs supported integrated care delivery, including transitions, documentation, information sharing, and decision making. We analyzed these data and employed a user-centered design process to develop EHR tools addressing the identified needs.RESULTS:BHCs used the primary care EHR for documentation and communication with other team members, but the EHR lacked the functionality to fully support integrated care. Needs include the ability to: (1) automate and track paper-based screening; (2) document behavioral health history; (3) access patient social and medical history relevant to behavioral health issues, and (4) rapidly document and track progress on goals. To meet these needs, we engaged users and developed a set of EHR tools called the BH e-Suite.CONCLUSION:Integrated primary care teams, and particularly BHCs, have unique information needs, workflows and tasks. These needs can be met and supported by the EHR with a moderate level of modification.

A national analysis of Asian Americans and Pacific Islanders (AAPI) substance use treatment admissions has yet to be studied. We sought to explore admission trends for AAPI, demographic and treatment variable change, and individual state admission change over time.We used retrospective time-series logistic regression treating year as a predictor of yearly treatment admission trends, between-state test for heterogeneity of treatment effects among states' AAPI admissions, and percent-changes of AAPI demographic and treatment characteristics. Participants included AAPI (n = 135,356) and comparison non-AAPI (n = 8,938,982) treatment groups' first-time admissions (N = 9,074,338) in United States treatment centers receiving public funding from 2000 to 2012.AAPI demonstrated a greater increase in admissions than non-AAPI from 2000 to 2012 (p < 0.0001; OR = 1.02, 95% CI = 1.019–1.022). Large percent increases were demonstrated in multiple demographic and treatment characteristic, most notably in prescription opioids as a problem substance, age of first use for the oldest and youngest groups, and homelessness. In addition, trends are provided for individual states to help prioritize resource need.The present demographic and treatment characteristics revealed specific variables that may help to improve a culturally competent understanding of increasing risk factors among AAPI clients. The present findings may help to demonstrate which states may need to increase AAPI-specific resources and interventions.

We define "Big Networks" as those that generate big data and can benefit from big data management in their operations. Examples of big networks include the current Internet and the emerging Internet of things and social networks. The ever-increasing scale, complexity and heterogeneity of the Internet make it harder to discover emergent and anomalous behavior in the network traffic. We hypothesize that endowing the otherwise semantically-oblivious Internet with "memory" management mimicking the human memory functionalities would help advance the Internet capability to learn, conceptualize and effectively and efficiently store traffic data and behavior, and to more accurately predict future events. Inspired by the functionalities of human memory, we proposed a distributed network memory management system, termed NetMem, to efficiently store Internet data and extract and utilize traffic semantics in matching and prediction processes. In particular, we explore Hidden Markov Models (HMM), Latent Dirichlet Allocation (LDA), and simple statistical analysis-based techniques for semantic reasoning in NetMem. Additionally, we propose a hybrid intelligence technique for semantic reasoning integrating LDA and HMM to extract network semantics based on learning patterns and features with syntax and semantic dependencies. We also utilize locality sensitive hashing for reducing dimensionality. Our simulation study using real network traffic demonstrates the benefits of NetMem and highlights the advantages and limitations of the aforementioned techniques.

IntroductionIt is not clear how to effectively recruit healthy research volunteers.MethodsWe developed an electronic health record (EHR)-based algorithm to identify healthy subjects, who were randomly assigned to receive an invitation to join a research registry via the EHR's patient portal, letters, or phone calls. A follow-up survey assessed contact preferences.ResultsThe EHR algorithm accurately identified 858 healthy subjects. Recruitment rates were low, but occurred more quickly via the EHR patient portal than letters or phone calls (2.7 vs. 19.3 or 10.4 d). Effort and costs per enrolled subject were lower for the EHR patient portal (3.0 vs. 17.3 or 13.6 h, $113 vs. $559 or $435). Most healthy subjects indicated a preference for contact via electronic methods.ConclusionsHealthy subjects can be accurately identified from EHR data, and it is faster and more cost-effective to recruit healthy research volunteers using an EHR patient portal.

INTRODUCTION:Brief smoking-cessation interventions in primary care settings are effective, but delivery of these services remains low. The Centers for Medicare and Medicaid Services' Meaningful Use (MU) of Electronic Health Record (EHR) Incentive Program could increase rates of smoking assessment and cessation assistance among vulnerable populations. This study examined whether smoking status assessment, cessation assistance, and odds of being a current smoker changed after Stage 1 MU implementation.METHODS:EHR data were extracted from 26 community health centers with an EHR in place by June 15, 2009. AORs were computed for each binary outcome (smoking status assessment, counseling given, smoking-cessation medications ordered/discussed, current smoking status), comparing 2010 (pre-MU), 2012 (MU preparation), and 2014 (MU fully implemented) for pregnant and non-pregnant patients.RESULTS:Non-pregnant patients had decreased odds of current smoking over time; odds for all other outcomes increased except for medication orders from 2010 to 2012. Among pregnant patients, odds of assessment and counseling increased across all years. Odds of discussing or ordering of cessation medications increased from 2010 compared with the other 2 study years; however, medication orders alone did not change over time, and current smoking only decreased from 2010 to 2012. Compared with non-pregnant patients, a lower percentage of pregnant patients were provided counseling.CONCLUSIONS:Findings suggest that incentives for MU of EHRs increase the odds of smoking assessment and cessation assistance, which could lead to decreased smoking rates among vulnerable populations. Continued efforts for provision of cessation assistance among pregnant patients is warranted.

OBJECTIVE:Variations in processes for different clinics and health systems can dramatically change the way preventive interventions are implemented. We present a method for documenting these variations using workflow diagrams and demonstrate how understanding workflow aided an electronic health record (EHR) embedded colorectal cancer screening intervention.MATERIALS AND METHODS:We mapped variation in processes for ordering and documenting fecal testing, current colonoscopy, prior colonoscopies, and pathology results. This work was part of a multi-site cluster-randomized pragmatic trial to test a mailed approach to offering fecal testing at 26 safety net clinics (in eight organizations) in Oregon and Northern California. We created clinic-specific workflow diagrams and then distilled them into consolidated diagrams that captured the variations.RESULTS:Clinics had varied practices for storing and using information about colorectal cancer screening. Developing workflow diagrams of key processes enabled clinics to find optimal ways to send fecal test kits to patients due for screening. The workflows informed the rollout of new EHR tools and identified best practices for data capture.DISCUSSION:Diagramming workflows can have great utility when implementing and refining EHR tools for clinical practice, especially when doing so across multiple clinical sites. The process of developing the workflows uncovered successful practice recommendations and revealed limitations and potential effects of a research intervention.CONCLUSION:Our method of documenting clinical process variation might inform other EHR-powered, multi-site research and can improve data feedback from EHR systems to clinical caregivers.

Summary Objective: Screening, brief intervention, and referral for treatment (SBIRT) for behavioral health (BH) is a key clinical process. SBIRT tools in electronic health records (EHR) are infrequent and rarely studied. Our goals were 1) to design and implement SBIRT using clinical decision support (CDS) in a commercial EHR; and 2) to conduct a pragmatic evaluation of the impact of the tools on clinical outcomes. Methods: A multidisciplinary team designed SBIRT workflows and CDS tools. We analyzed the outcomes using a retrospective descriptive convenience cohort with age-matched comparison group. Data extracted from the EHR were evaluated using descriptive statistics. Results: There were 2 outcomes studied: 1) development and use of new BH screening tools and workflows; and 2) the results of use of those tools by a convenience sample of 866 encounters. The EHR tools developed included a flowsheet for documenting screens for 3 domains (depression, alcohol use, and prescription misuse); and 5 alerts with clinical recommendations based on screening; and reminders for annual screening. Positive screen rate was 21% (≥1 domain) with 60% of those positive for depression. Screening was rarely positive in 2 domains (11%), and never positive in 3 domains. Positive and negative screens led to higher rates of documentation of brief intervention (BI) compared with a matched sample who did not receive screening, including changes in psychotropic medications, updated BH terms on the problem list, or referral for BH intervention. Clinical process outcomes changed even when screening was negative. Conclusions: Modified workflows for BH screening and CDS tools with clinical recommendations can be deployed in the EHR. Using SBIRT tools changed clinical process metrics even when screening was negative, perhaps due to conversations about BH not captured in the screening flowsheet. Although there are limitations to the study, results support ongoing investigation. Citation: Burdick TE, Kessler RS. Development and use of a clinical decision support tool for behavioral health screening in primary care clinics. Appl Clin Inform 2017; 8: 412–429 https://doi.org/10.4338/ACI-2016-04-RA-0068