AbstractObjectiveTo evaluate the validity of death ascertainment from publicly available internet media (IM) sources by benchmarking against state and Federal vital statics data for patients in two large healthcare systems from the US.MethodsWe extracted names and dates of birth and death from publicly available data—including obituaries and memorial websites—using previously developed natural language processing models. These data were probabilistically matched to electronic health records (EHRs) from Mass General Brigham (MGB) and Vanderbilt University Medical Center (VUMC) on first name, last name, and date of birth. Using reference standards from state vital statistics databases from MA, CT, and VT for MGB and the National Death Index (NDI) for VUMC patients, we reported positive predicted values (PPV) considering cases where dates of death from IM sources were within 7 days of the reference standard to be true positives. We also reported sensitivity of deaths ascertained from IM sources.ResultsWhen probabilistically matching 8.1 million deaths extracted from public data to 78,848 deaths observed in the reference standards across two sites, 30,607 (38.8%) matched exactly. A PPV of 98.2% for MGB and 98.9% for VUMC was observed for exact matches, while <6% for non-exact matches. Considering only the exact matches, IM sources led to an improvement in sensitivity of death capture by 24% in MGB and 18% in VUMC, compared to using EHRs alone for death ascertainment.ConclusionsUsing public information to augment mortality data increased capture of death meaningfully over reliance on EHR records alone.

Objectives: Traditional methods for medical device post-market surveillance often fail to accurately account for operator learning effects, leading to biased assessments of device safety. These methods struggle with non-linearity, complex learning curves, and time-varying covariates, such as physician experience. To address these limitations, we sought to develop a machine learning (ML) framework to detect and adjust for operator learning effects. Materials and Methods: A gradient-boosted decision tree ML method was used to analyze synthetic datasets that replicate the complexity of clinical scenarios involving high-risk medical devices. We designed this process to detect learning effects using a risk-adjusted cumulative sum method, quantify the excess adverse event rate attributable to operator inexperience, and adjust for these alongside patient factors in evaluating device safety signals. To maintain integrity, we employed blinding between data generation and analysis teams. Synthetic data used underlying distributions and patient feature correlations based on clinical data from the Department of Veterans Affairs between 2005 and 2012. We generated 2494 synthetic datasets with widely varying characteristics including number of patient features, operators and institutions, and the operator learning form. Each dataset contained a hypothetical study device, Device B, and a reference device, Device A. We evaluated accuracy in identifying learning effects and identifying and estimating the strength of the device safety signal. Our approach also evaluated different clinically relevant thresholds for safety signal detection. Results:Our framework accurately identified the presence or absence of learning effects in 93.6% of datasets and correctly determined device safety signals in 93.4% of cases. The estimated device odds ratios' 95% confidence intervals were accurately aligned with the specified ratios in 94.7% of datasets. In contrast, a comparative model excluding operator learning effects significantly underperformed in detecting device signals and in accuracy. Notably, our framework achieved 100% specificity for clinically relevant safety signal thresholds, although sensitivity varied with the threshold applied. Discussion: A machine learning framework, tailored for the complexities of post-market device evaluation, may provide superior performance compared to standard parametric techniques when operator learning is present. Conclusion: Demonstrating the capacity of ML to overcome complex evaluative challenges, our framework addresses the limitations of traditional statistical methods in current post-market surveillance processes. By offering a reliable means to detect and adjust for learning effects, it may significantly improve medical device safety evaluation.

Background: Mortality is a critical variable in healthcare research, especially for evaluating medical product safety and effectiveness. However, inconsistencies in the availability and timeliness of death date and cause of death (CoD) information present significant challenges. Conventional sources such as the National Death Index (NDI) and electronic health records (EHRs) often suffer from data lags, missing fields, or incomplete coverage, limiting their utility in time-sensitive or large-scale studies. With the growing use of social media, crowdfunding platforms, and online memorials, publicly available digital content has emerged as a potential supplementary source for mortality surveillance. Despite this potential, accurate tools for extracting mortality information from such unstructured data sources remain underdeveloped. Objective: To develop scalable approaches using natural language processing (NLP) and large language models (LLM) for the extraction of mortality information from publicly available online data sources, including social media platforms, crowdfunding websites, and online obituaries, and to evaluate their performance across various sources. Methods. Data were collected from public posts on X (formerly Twitter), GoFundMe campaigns, memorial websites (EverLoved.com and TributeArchive.com), and online obituaries from 2015 to 2022, focusing on U.S.-based content relevant to mortality. We developed an NLP pipeline using transformer-based models to extract key mortality information such as decedent names, dates of birth, and dates of death. We then employed a few-shot learning (FSL) approach with LLMs to identify primary and secondary causes of death. Model performance was assessed using precision, recall, F1-score, and accuracy metrics, with human-annotated labels serving as the reference standard for the transformer-based model and a human adjudicator blinded to labeling source for the FSL model reference standard. Results: The best-performing model obtained a micro-averaged F1-score of 0.88 (95% CI, 0.86-0.90) in extracting mortality information. The FSL-LLM approach demonstrated high accuracy in identifying primary CoD across various online sources. For GoFundMe, the FSL-LLM achieved 95.9% accuracy for primary cause identification, compared to 97.9% for human annotators. In obituaries, FSL-LLM accuracy was 96.5% for primary causes, while human accuracy was 99.0%. For memorial websites, FSL-LLM achieved 98.0% accuracy for primary causes, with human accuracy at 99.5%. Conclusions: This study demonstrates the feasibility of using advanced NLP and LLM techniques to extract mortality data from publicly available online sources. These methods can significantly enhance the timeliness, completeness, and granularity of mortality surveillance, offering a valuable complement to traditional data systems. By enabling earlier detection of mortality signals and improving CoD classification across large populations, this approach may support more responsive public health monitoring and medical product safety assessments. Further work is needed to validate these findings in real-world healthcare settings and facilitate the integration of digital data sources into national public health surveillance systems. ### Competing Interest Statement The authors have declared no competing interest. ### Funding Statement Funding statement: This project was supported by Task Order 75F40123F19010 under Master Agreement 75F40119D10037 from the US Food and Drug Administration (FDA). FDA coauthors reviewed the study protocol, statistical analysis plan, and the manuscript for scientific accuracy and clarity of presentation. Representatives of the FDA reviewed a draft of the manuscript for presence of confidential information and accuracy regarding statement of any FDA policy. The views expressed are those of the authors and not necessarily those of the US FDA. ### Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes All data produced in the present study are available upon reasonable request to the authors

Background Validating new algorithms, such as methods to disentangle intrinsic treatment risk from risk associated with experiential learning of novel treatments, often requires knowing the ground truth for data characteristics under investigation. Since the ground truth is inaccessible in real world data, simulation studies using synthetic datasets that mimic complex clinical environments are essential. We describe and evaluate a generalizable framework for injecting hierarchical learning effects within a robust data generation process that incorporates the magnitude of intrinsic risk and accounts for known critical elements in clinical data relationships. Methods We present a multi-step data generating process with customizable options and flexible modules to support a variety of simulation requirements. Synthetic patients with nonlinear and correlated features are assigned to provider and institution case series. The probability of treatment and outcome assignment are associated with patient features based on user definitions. Risk due to experiential learning by providers and/or institutions when novel treatments are introduced is injected at various speeds and magnitudes. To further reflect real-world complexity, users can request missing values and omitted variables. We illustrate an implementation of our method in a case study using MIMIC-III data for reference patient feature distributions. Results Realized data characteristics in the simulated data reflected specified values. Apparent deviations in treatment effects and feature distributions, though not statistically significant, were most common in small datasets (n < 3000) and attributable to random noise and variability in estimating realized values in small samples. When learning effects were specified, synthetic datasets exhibited changes in the probability of an adverse outcomes as cases accrued for the treatment group impacted by learning and stable probabilities as cases accrued for the treatment group not affected by learning. Conclusions Our framework extends clinical data simulation techniques beyond generation of patient features to incorporate hierarchical learning effects. This enables the complex simulation studies required to develop and rigorously test algorithms developed to disentangle treatment safety signals from the effects of experiential learning. By supporting such efforts, this work can help identify training opportunities, avoid unwarranted restriction of access to medical advances, and hasten treatment improvements.

HomeCirculationVol. 147, No. 18Cluster Randomized Trial of a Personalized Clinical Decision Support Intervention to Improve Statin Prescribing in Patients With Atherosclerotic Cardiovascular Disease Free AccessLetterPDF/EPUBAboutView PDFView EPUBSections ToolsAdd to favoritesDownload citationsTrack citationsPermissions ShareShare onFacebookTwitterLinked InMendeleyReddit Jump toFree AccessLetterPDF/EPUBCluster Randomized Trial of a Personalized Clinical Decision Support Intervention to Improve Statin Prescribing in Patients With Atherosclerotic Cardiovascular Disease Salim S. Virani, David J. Ramsey, Dax Westerman, Mark K. Kuebeler, Liang Chen, Julia M. Akeroyd, Glenn T. Gobbel, Christie M. Ballantyne, Laura A. Petersen, Alexander Turchin and Michael E. Matheny Salim S. ViraniSalim S. Virani Correspondence to: Salim S. Virani, MD, PhD, Health Services Research and Development, Michael E. DeBakey Veterans Affairs Medical Center, 2002 Holcombe Blvd, Houston, TX 77030. Email E-mail Address: [email protected] https://orcid.org/0000-0001-9541-6954 Health Policy, Quality & Informatics Program, Michael E. DeBakey Veterans Affairs Medical Center, Health Services Research & Development Center for Innovations in Quality, Effectiveness, and Safety, Houston, TX (S.S.V., D.J.R., M.K.K., J.M.A., L.A.P.). Section of Cardiovascular Research (S.S.V., C.M.B.), Department of Medicine, Baylor College of Medicine, Houston, TX. The Aga Khan University, Karachi, Pakistan (S.S.V.). , David J. RamseyDavid J. Ramsey Health Policy, Quality & Informatics Program, Michael E. DeBakey Veterans Affairs Medical Center, Health Services Research & Development Center for Innovations in Quality, Effectiveness, and Safety, Houston, TX (S.S.V., D.J.R., M.K.K., J.M.A., L.A.P.). , Dax WestermanDax Westerman https://orcid.org/0000-0002-9547-7789 Geriatrics Research Education and Clinical Care, Tennessee Valley Healthcare System VA, Nashville (D.W., G.T.G., M.E.M.). Departments of Biomedical Informatics, Biostatistics, and Medicine, Vanderbilt University Medical Center, Nashville, TN (D.W., G.T.G., M.E.M.). , Mark K. KuebelerMark K. Kuebeler Health Policy, Quality & Informatics Program, Michael E. DeBakey Veterans Affairs Medical Center, Health Services Research & Development Center for Innovations in Quality, Effectiveness, and Safety, Houston, TX (S.S.V., D.J.R., M.K.K., J.M.A., L.A.P.). , Liang ChenLiang Chen https://orcid.org/0000-0001-7395-3759 , Julia M. AkeroydJulia M. Akeroyd Health Policy, Quality & Informatics Program, Michael E. DeBakey Veterans Affairs Medical Center, Health Services Research & Development Center for Innovations in Quality, Effectiveness, and Safety, Houston, TX (S.S.V., D.J.R., M.K.K., J.M.A., L.A.P.). , Glenn T. GobbelGlenn T. Gobbel Geriatrics Research Education and Clinical Care, Tennessee Valley Healthcare System VA, Nashville (D.W., G.T.G., M.E.M.). Departments of Biomedical Informatics, Biostatistics, and Medicine, Vanderbilt University Medical Center, Nashville, TN (D.W., G.T.G., M.E.M.). , Christie M. BallantyneChristie M. Ballantyne https://orcid.org/0000-0002-6432-1730 Section of Cardiovascular Research (S.S.V., C.M.B.), Department of Medicine, Baylor College of Medicine, Houston, TX. , Laura A. PetersenLaura A. Petersen Health Policy, Quality & Informatics Program, Michael E. DeBakey Veterans Affairs Medical Center, Health Services Research & Development Center for Innovations in Quality, Effectiveness, and Safety, Houston, TX (S.S.V., D.J.R., M.K.K., J.M.A., L.A.P.). Section of Health Services Research (L.A.P.), Department of Medicine, Baylor College of Medicine, Houston, TX. , Alexander TurchinAlexander Turchin https://orcid.org/0000-0002-8609-564X Harvard Medical School, Boston, MA (A.T.). Division of Endocrinology, Brigham and Women's Hospital, Boston, MA (A.T.). and Michael E. MathenyMichael E. Matheny https://orcid.org/0000-0003-3217-4147 Geriatrics Research Education and Clinical Care, Tennessee Valley Healthcare System VA, Nashville (D.W., G.T.G., M.E.M.). Departments of Biomedical Informatics, Biostatistics, and Medicine, Vanderbilt University Medical Center, Nashville, TN (D.W., G.T.G., M.E.M.). Originally published5 Mar 2023https://doi.org/10.1161/CIRCULATIONAHA.123.064226Circulation. 2023;147:1411–1413Other version(s) of this articleYou are viewing the most recent version of this article. Previous versions: March 5, 2023: Ahead of Print Statin and high-intensity statin (HIS) use remains low in patients with atherosclerotic cardiovascular disease (ASCVD).1,2 Both statin-associated side effects (SASEs) and therapeutic inertia play a role.1 We evaluated whether personalized reminders improve HIS use in patients with ASCVD.In this cluster-randomized controlled trial performed in the Department of Veterans Affairs, we tested an intervention developed over 4 years by constructing algorithms using structured3 and unstructured data through natural language processing4 to identify SASEs and, by performing qualitative interviews, to understand patient perspectives on SASEs and clinician information needs.5 Leveraging this, an intervention was developed that included reminders processed by the research team at one location individualized to each patient and sent to their respective primary care clinicians 2 to 7 days before their next visit (synchronous reminders) or outside of the primary care visit (asynchronous reminders). Information on reminders included date and type of ASCVD diagnosis (ischemic heart disease, peripheral artery disease, or ischemic stroke), statin and dose, date of last fill, date and type of SASE, and guideline resources on HIS definition and SASE management. To prevent alert fatigue, our algorithms ensured that clinicians did not have >3 unsigned alerts from this reminder before receiving more reminders. Clinicians at the intervention sites could opt out from receiving reminders. Usual care included clinician access to a patient dashboard displaying compliance with statin therapy. The cohort was updated monthly to incorporate updated statin dose, new exclusions (metastatic cancer, hospice, palliative care, or death), and new SASEs. Ethical approval was obtained from the institutional ethics review committee. Study data are available from the corresponding author on reasonable request.Following guideline education, we randomly assigned 27 primary care clinics (36 641 patients): 14 to the intervention (117 clinicians and 18 427 patients) and 13 to usual care (128 clinicians and 18 214 patients). Outcomes included before and after changes in HIS (primary) and statin (secondary) use between intervention and usual care sites. Because we expected that the reminder would allow clinicians to elicit patients' concerns about statins, we evaluated before and after statin adherence (using proportion of day covered ≥0.8) difference between 2 groups. The trial began in August 2021 and ended in November 2022.Mean age was 71.1 years, and the predominant ASCVD phenotype included ischemic heart disease (77.5% patients). Of the patients in the intervention arm, 41.6% had a signal related to SASEs on either structured data or natural language processing.We sent 4928 reminders (73% asynchronous and 27% synchronous) for 4532 unique patients, representing 53% of the patients not on HIS at baseline in the intervention arm. Throughout the study, 37 clinicians (31.6%) at the intervention sites opted out. Baseline HIS use at intervention and usual care sites was 53.6% and 55.9%, respectively. At the end of the study, HIS use at intervention and usual care sites was 55.2% and 53.7%, respectively (between-group ∆=3.8% [95% CI, 3.7%–3.9%]; odds ratio for HIS use with the intervention, 1.06 [95% CI, 1.02–1.11]). In the intervention arm, the absolute change in HIS was +10.1% for those who received a reminder versus a –0.18% decrease among those who did not receive a reminder (Figure). In patients on whom a reminder was sent, 11.6% initiated HIS with synchronous compared with 9.6% with asynchronous reminders (P=0.58). Among those receiving a reminder, the increase in HIS was 9.1% and 10.9% among those with or without SASEs, respectively (P=0.02).Download figureDownload PowerPointFigure. Before and after change in high-intensity statin therapy use in patients who have atherosclerotic cardiovascular disease receiving care at usual care and intervention sites (overall, among those who did not receive reminders, and among those who received reminders).Statin use decreased at both intervention (81.1% to 78.7%) and usual care sites (82% to 76.8%) with a between-group ∆ of 2.8% in favor of the intervention arm ([95% CI, 2.7%–2.9%]; odds ratio, 1.12 [95% CI,1.06–1.18]). In the intervention arm, before and after statin use remained the same (+0.04%) for those who received a reminder versus a 3.14% decrease for those who did not receive a reminder. The number of patients with proportion of day covered ≥0.8 increased in the intervention arm from 60.8% to 67.2% and from 54.4% to 58% in the usual care arm, with a between-group difference of 2.8% (odds ratio for proportion of day covered ≥0.8, 1.38 [95% CI, 1.32–1.45]).In this multisite randomized controlled trial, centrally processed individualized reminders led to a significant increase in HIS use and statin adherence in patients with ASCVD. Although overall effect size was modest, it was much higher among 53% of eligible patients for whom a reminder was sent (≈10 reminders needed to be sent for 1 patient to be initiated on HIS). Reminders were not sent on all eligible patients because of stringent algorithms to limit alert fatigue. A low number of synchronous reminders, clinician dropout over time, lack of a clinical decision support algorithm embedded within the intervention, and cognitive demands on clinician time due to iterative coronavirus disease 2019 (COVID-19) surges during the study may also have reduced the overall intervention efficacy.To our knowledge, this is the first multisite study leveraging structured data and natural language processing to provide individualized information to clinicians to improve HIS use in patients with ASCVD. Centrally controlled reminders allowed rapid upscaling of intervention to several sites. Our future work will focus on actions taken by clinicians in response to reminders and barriers and facilitators to our intervention. Our study results inform how informatics-driven interventions can improve evidence-based care delivery in large health care systems.Article InformationSources of FundingThis work was supported by a Department of Veterans Affairs (VA) Health Services Research & Development service investigator initiated grant (IIR 16-072) and a Houston VA Health Services Research & Development Center for Innovations grant (CIN13413). Support for VA/Centers for Medicare & Medicaid Services data was provided by the Department of Veterans Affairs, VA Health Services Research and Development Service, VA Information Resource Center.Nonstandard Abbreviations and AcronymsASCVDatherosclerotic cardiovascular diseaseHIShigh-intensity statin therapySASEstatin-associated side effectDisclosures Dr Virani is supported by research grants from the Department of Veterans Affairs, the National Institutes of Health (NIH), Tahir and Jooma Family; and has received honoraria from the American College of Cardiology in his role as associate editor for Innovations, ACC.org. Dr Turchin has received research support from Astra Zeneca, Eli Lilly, Novo Nordisk; consulting fees from Novo Nordisk, Proteomics International; and holds equity interest in Brio Systems. Dr Ballantyne has received grants/research support from Abbott Diagnostic, Akcea, Amgen, Arrowhead, Esperion, Ionis, Merck, Novartis, Novo Nordisk, Regeneron, Roche Diagnostic, NIH, American Heart Association, and Americans With Disabilities Act[AQ: Please confirm expansion of ADA in Disclosures.]; and consulting fees from 89Bio, Abbott Diagnostics, Alnylam Pharmaceuticals, Althera, Amarin, Amgen, Arrowhead, Astra Zeneca, Denka Seiken, Esperion, Genentech, Gilead, Illumina, Ionis, Matinas BioPharma Inc, Merck, New Amsterdam, Novartis, Novo Nordisk, Pfizer, Regeneron, and Roche Diagnostic.FootnotesThe opinions expressed reflect those of the authors and not necessarily those of the Department of Veterans Affairs or the US government.For Sources of Funding and Disclosures, see page 1413.Circulation is available at www.ahajournals.org/journal/circCorrespondence to: Salim S. Virani, MD, PhD, Health Services Research and Development, Michael E. DeBakey Veterans Affairs Medical Center, 2002 Holcombe Blvd, Houston, TX 77030. Email virani@bcm.eduReferences1. Navar AM, Wang TY, Li S, Robinson JG, Goldberg AC, Virani S, Roger VL, Wilson PWF, Elassal J, Lee LV, et al. Lipid management in contemporary community practice: results from the Provider Assessment of Lipid Management (PALM) Registry.Am Heart J. 2017; 193:84–92. doi: 10.1016/j.ahj.2017.08.005CrossrefMedlineGoogle Scholar2. Virani SS, Ballantyne CM, Petersen LA. Guideline-concordant statin therapy use in secondary prevention: should the medical community wait for divine intervention?J Am Coll Cardiol. 2022; 79:1814–1817. doi: 10.1016/j.jacc.2022.02.042CrossrefMedlineGoogle Scholar3. Virani SS, Akeroyd JM, Ahmed ST, Krittanawong C, Martin LA, Slagle J, Gobbel GT, Matheny ME, Ballantyne CM, Petersen LA. The use of structured data elements to identify ASCVD patients with statin-associated side effects: Insights from the Department of Veterans Affairs.J Clin Lipidol. 2019; 13:797–803.e1. doi: 10.1016/j.jacl.2019.08.002CrossrefMedlineGoogle Scholar4. Gobbel GT, Matheny ME, Reeves RR, Akeroyd JM, Turchin A, Ballantyne CM, Petersen LA, Virani SS. Leveraging structured and unstructured electronic health record data to detect reasons for suboptimal statin therapy use in patients with atherosclerotic cardiovascular disease.Am J Prev Cardiol. 2021; 9:100300. doi: 10.1016/j.ajpc.2021.100300CrossrefMedlineGoogle Scholar5. Ahmed ST, Akeroyd JM, Mahtta D, Street R, Slagle J, Navar AM, Stone NJ, Ballantyne CM, Petersen LA, Virani SS. Shared decisions: a qualitative study on clinician and patient perspectives on statin therapy and statin-associated side effects.J Am Heart Assoc. 2020; 9:e017915. doi: 10.1161/JAHA.120.017915LinkGoogle Scholar eLetters(0) eLetters should relate to an article recently published in the journal and are not a forum for providing unpublished data. Comments are reviewed for appropriate use of tone and language. Comments are not peer-reviewed. Acceptable comments are posted to the journal website only. Comments are not published in an issue and are not indexed in PubMed. Comments should be no longer than 500 words and will only be posted online. References are limited to 10. Authors of the article cited in the comment will be invited to reply, as appropriate. Comments and feedback on AHA/ASA Scientific Statements and Guidelines should be directed to the AHA/ASA Manuscript Oversight Committee via its Correspondence page. Sign In to Submit a Response to This Article Previous Back to top Next FiguresReferencesRelatedDetailsCited By Karalis D (2023) Strategies of improving adherence to lipid-lowering therapy in patients with atherosclerotic cardiovascular disease, Current Opinion in Lipidology, 10.1097/MOL.0000000000000896, 34:6, (252-258), Online publication date: 1-Dec-2023. Wilkinson M, Lepor N and Michos E (2023) Evolving Management of Low‐Density Lipoprotein Cholesterol: A Personalized Approach to Preventing Atherosclerotic Cardiovascular Disease Across the Risk Continuum, Journal of the American Heart Association, 12:11, Online publication date: 6-Jun-2023. Gupta K, Balachandran I, Foy J, Hermel M, Latif A, Krittanawong C, Slipczuk L, Baloch F, Samad Z and Virani S (2023) Highlights of Cardiovascular Disease Prevention Studies Presented at the 2023 American College of Cardiology Conference, Current Atherosclerosis Reports, 10.1007/s11883-023-01103-4, 25:6, (309-321), Online publication date: 1-Jun-2023. May 2, 2023Vol 147, Issue 18 Advertisement Article Information Metrics © 2023 American Heart Association, Inc.https://doi.org/10.1161/CIRCULATIONAHA.123.064226PMID: 36871214 Originally publishedMarch 5, 2023 Keywordsadverse effectsatherosclerosisdata sciencehydroxymethylglutaryl-CoA reductase inhibitorsPDF download Advertisement Subjects Cardiovascular Disease Health Services Quality and Outcomes Risk Factors Secondary Prevention

Background Up to 14% of patients in the United States undergoing cardiac catheterization each year experience AKI. Consistent use of risk minimization preventive strategies may improve outcomes. We hypothesized that team-based coaching in a Virtual Learning Collaborative (Collaborative) would reduce postprocedural AKI compared with Technical Assistance (Assistance), both with and without Automated Surveillance Reporting (Surveillance). Methods The IMPROVE AKI trial was a 2x2 factorial cluster-randomized trial across 20 Veterans Affairs medical centers (VAMCs). Participating VAMCs received Assistance, Assistance with Surveillance, Collaborative, or Collaborative with Surveillance for 18 months to implement AKI prevention strategies. The Assistance and Collaborative approaches promoted hydration and limited NPO and contrast dye dosing. We fit logistic regression models for AKI with site-level random effects accounting for the clustering of patients within medical centers with a prespecified interest in exploring differences across the four intervention arms. Results Among VAMCs' 4517 patients, 510 experienced AKI (235 AKI events among 1314 patients with preexisting CKD). AKI events in each intervention cluster were 110 (13%) in Assistance, 122 (11%) in Assistance with Surveillance, 190 (13%) in Collaborative, and 88 (8%) in Collaborative with Surveillance. Compared with sites receiving Assistance alone, case-mix-adjusted differences in AKI event proportions were -3% (95% confidence interval [CI], -4 to -3) for Assistance with Surveillance, -3% (95% CI, -3 to -2) for Collaborative, and -5% (95% CI, -6 to -5) for Collaborative with Surveillance. The Collaborative with Surveillance intervention cluster had a substantial 46% reduction in AKI compared with Assistance alone (adjusted odds ratio=0.54; 0.40-0.74). Conclusions This implementation trial estimates that the combination of Collaborative with Surveillance reduced the odds of AKI by 46% at VAMCs and is suggestive of a reduction among patients with CKD.

Background: The utility of quality dashboards to inform decision-making and improve clinical outcomes is tightly linked to the accuracy of the information they provide and, in turn, accuracy of underlying prediction models. Despite recognition of the need to update prediction models to maintain accuracy over time, there is limited guidance on updating strategies. We compare predefined and surveillance-based updating strategies applied to a model supporting quality evaluations among US veterans. Methods: We evaluated the performance of a US Department of Veterans Affairs–specific model for postcardiac catheterization acute kidney injury using routinely collected observational data over the 6 years following model development (n=90 295 procedures in 2013–2019). Predicted probabilities were generated from the original model, an annually retrained model, and a surveillance-based approach that monitored performance to inform the timing and method of updates. We evaluated how updating the national model impacted regional quality profiles. We compared observed-to-expected outcome ratios, where values above and below 1 indicated more and fewer adverse outcomes than expected, respectively. Results: The original model overpredicted risk at the national level (observed-to-expected outcome ratio, 0.75 [0.74–0.77]). Annual retraining updated the model 5×; surveillance-based updating retrained once and recalibrated twice. While both strategies improved performance, the surveillance-based approach provided superior calibration (observed-to-expected outcome ratio, 1.01 [0.99–1.03] versus 0.94 [0.92–0.96]). Overprediction by the original model led to optimistic quality assessments, incorrectly indicating most of the US Department of Veterans Affairs’ 18 regions observed fewer acute kidney injury events than predicted. Both updating strategies revealed 16 regions performed as expected and 2 regions increasingly underperformed, having more acute kidney injury events than predicted. Conclusions: Miscalibrated clinical prediction models provide inaccurate pictures of performance across clinical units, and degrading calibration further complicates our understanding of quality. Updating strategies tailored to health system needs and capacity should be incorporated into model implementation plans to promote the utility and longevity of quality reporting tools.

BACKGROUND:Despite its high prevalence and clinical impact, research on peripheral artery disease (PAD) remains limited due to poor accuracy of billing codes. Ankle-brachial index (ABI) and toe-brachial index can be used to identify PAD patients with high accuracy within electronic health records. METHODS:We developed a novel natural language processing (NLP) algorithm for extracting ABI and toe-brachial index values and laterality (right or left) from ABI reports. A random sample of 800 reports from 94 Veterans Affairs facilities during 2015 to 2017 was selected and annotated by clinical experts. We trained the NLP system using random forest models and optimized it through sequential iterations of 10-fold cross-validation and error analysis on 600 test reports and evaluated its final performance on a separate set of 200 reports. We also assessed the accuracy of NLP-extracted ABI and toe-brachial index values for identifying patients with PAD in a separate cohort undergoing ABI testing. RESULTS:The NLP system had an overall precision (positive predictive value) of 0.85, recall (sensitivity) of 0.93, and F1 measure (accuracy) of 0.89 to correctly identify ABI/toe-brachial index values and laterality. Among 261 patients with ABI testing (49% PAD), the NLP system achieved a positive predictive value of 92.3%, sensitivity of 83.1%, and specificity of 93.1% to identify PAD when compared with a structured chart review. The above findings were consistent in a range of sensitivity analysis. CONCLUSIONS:We successfully developed and validated an NLP system for identifying patients with PAD within the Veterans Affairs electronic health record. Our findings have broad implications for PAD research and quality improvement.

BACKGROUND There are gaps in delivering evidence-based care for patients with chronic liver disease and cirrhosis. OBJECTIVE Our objective was to use interactive user-centered design methods to develop the Cirrhosis Order Set and Clinical Decision Support (CirrODS) tool in order to improve clinical decision-making and workflow. METHODS Two work groups were convened with clinicians, user experience designers, human factors and health services researchers, and information technologists to create user interface designs. CirrODS prototypes underwent several rounds of formative design. Physicians (n=20) at three hospitals were provided with clinical scenarios of patients with cirrhosis, and the admission orders made with and without the CirrODS tool were compared. The physicians rated their experience using CirrODS and provided comments, which we coded into categories and themes. We assessed the safety, usability, and quality of CirrODS using qualitative and quantitative methods. RESULTS We created an interactive CirrODS prototype that displays an alert when existing electronic data indicate a patient is at risk for cirrhosis. The tool consists of two primary frames, presenting relevant patient data and allowing recommended evidence-based tests and treatments to be ordered and categorized. Physicians viewed the tool positively and suggested that it would be most useful at the time of admission. When using the tool, the clinicians placed fewer orders than they placed when not using the tool, but more of the orders placed were considered to be high priority when the tool was used than when it was not used. The physicians’ ratings of CirrODS indicated above average usability. CONCLUSIONS We developed a novel Web-based combined clinical decision-making and workflow support tool to alert and assist clinicians caring for patients with cirrhosis. Further studies are underway to assess the impact on quality of care for patients with cirrhosis in actual practice.