Utility of artificial intelligence-based tool for medical education during rounds in the icu

SESSION TITLE: Critical Care Posters 3 SESSION TYPE: Original Investigation Posters PRESENTED ON: 10/10/2023 12:00 pm - 12:45 pm PURPOSE: ChatGPT is a new AI technology launched by OpenAI on November 30, 2022. It has successfully written college essays and passed rigorous standardized tests at this point. There is still debate about whether this technology is reliable enough to be utilized as a medical education tool. This research aimed to determine the feasibility of ChatGPT to assist in the education of medical students, residents, fellows, and medical staff during rounds in the critical care setting where time is limited. METHODS: 100 questions were submitted to ChatGPT by 3 attendings with time and accuracy tabulated. Each set of questions submitted by each attending utilized a different question scheme: multiple choice, common questions asked on rounds, and questions based on published literature at least 2 years old. The same questions were answered by an additional 2 recent graduate attendings based on foundational knowledge without the use of resources and by 3 fellows based on online resources typically available on rounds. Accuracy and time were tabulated to compare against the AI output. Answers were graded as “correct”, “partially correct”, and “incorrect” by the original 3 attendings who wrote the 100 questions. RESULTS: Analysis showed that the AI answered 56% of questions accurately, 14% of answers were partially correct, and 30% were incorrect. The two attendings answered 78% of questions accurately, 5% were partially correct, and 14% were incorrect. The 3 fellows answered 83% of questions accurately, 1% were partially correct, and 11.66% were incorrect. In terms of time, the AI spent a mean of 23.35 seconds per question, attendings a mean of 8.44 seconds per question, and fellows a mean of 69.04 seconds per question. ANOVA analysis yielded a p-value <0.001. CONCLUSIONS: Although this new technology does hold the potential to be utilized as an educational tool as it is easy to use and quick to answer, the accuracy of information is not up to par when compared to currently available resources. It will be interesting to see the evolution of ChatGPT in newer versions (3.5, 4.0) and compare results in the future. CLINICAL IMPLICATIONS: At this time, the evolution of this technology needs to be followed carefully to see if it can be reliably used as an educational tool. If used in a clinical setting during rounds in the ICU setting, careful fact-checking must be done to ensure the accuracy of AI output. DISCLOSURES: No disclosure on file for Alejandro Arroliga No relevant relationships by Shreyes Boddu No disclosure on file for Joseph Carlin No relevant relationships by Shekhar Ghamande No relevant relationships by Tasnim Lat No relevant relationships by Eugene Quaye No relevant relationships by Zenia Sattar No relevant relationships by Munish Sharma No relevant relationships by Abirami Subramanian No relevant relationships by Heath White