Beyond Text: The Impact of Clinical Context on GPT-4’s 12-lead ECG Interpretation Accuracy

Abstract Introduction Artificial intelligence (AI) and large language models (LLMs), such as OpenAI's Chat Generative Pre-trained Transformer – version 4 (GPT-4), are being increasingly explored for medical applications, including clinical decision support. The introduction of the capability to analyze graphical inputs marks a significant advancement in the functionality of GPT-4. Despite the promising potential of AI in enhancing diagnostic accuracy, the effectiveness of GPT-4 in interpreting complex 12-lead electrocardiograms (ECGs) remains to be assessed. Methods This study utilized GPT-4 to interpret 150 12-lead ECGs from the Cardiology Research Dubrava (CaRD) registry, spanning a wide range of cardiac pathologies. The ECGs were classified into four categories for analysis: Arrhythmias (Category 1), Conduction System abnormalities (Category 2), Acute Coronary Syndrome (Category 3), and Other (Category 4). Two experiments were conducted: one where GPT-4 interpreted ECGs without clinical context and another with added clinical scenarios. A panel of experienced cardiologists evaluated the accuracy of GPT-4's interpretations. Statistical significance was determined using the Shapiro-Wilk test for distribution, Mann-Whitney U test for continuous variables, and Chi-square/Fisher's exact tests for categorical variables. Results In this cross-sectional, observational study, GPT-4 demonstrated a correct interpretation rate of 19% without clinical context and a significantly improved rate of 45% with context (p < 0.001). The addition of clinical scenarios significantly enhanced interpretative accuracy, particularly in the Category 3 (Acute Coronary Syndrome) (10 vs. 70%, p < 0.0.01). Unlike Category 4 (Other) which showed no impact (51 vs. 59%, p = 0.640), an impact with a trend toward significance was observed in Category 1 (Arrhythmias) (9.7 vs. 32%, p = 0.059) and Category 2 (Conduction System abnormalities) (4.8 vs. 19%, p = 0.088) when tasked with context. Conclusion While GPT-4 shows some potential in aiding ECG interpretation, its effectiveness varies significantly depending on the presence of clinical context. The study suggests that, in its current form, GPT-4 alone may not suffice for accurate ECG interpretation across a broad spectrum of cardiac conditions.