Evaluating General Vision-Language Models for Clinical Medicine

Recently emerging large multimodal models (LMMs) utilize various types of data modalities, including text and visual inputs to generate outputs. The incorporation of LMMs into clinical medicine presents unique challenges, including accuracy, reliability, and clinical relevance. Here, we explore clinical applications of GPT-4V, an LMM that has been proposed for use in medicine, in gastroenterology, radiology, dermatology, and United States Medical Licensing Examination (USMLE) test questions. We used standardized robust datasets with thousands of endoscopy images, chest x-ray, and skin lesions to benchmark GPT-4V's ability to predict diagnoses. To assess bias, we also explored GPT-4V's ability to determine Fitzpatrick skin tones with dermatology images. We found that GPT-4V is limited in performance across all four domains, resulting in decreased performance compared to previously-published baseline models. The macro-average precision, recall, and F1-score for gastroenterology was 11.2%, 9.1% and 6.8% respectively. For radiology, the best performing task of identifying cardiomegaly had precision, recall, and F1-score of 28%, 94%, and 43% respectively. In dermatology, GPT-4V had an overall top-1 and top-3 diagnostic accuracy of 6.2% and 21% respectively. There was a significant accuracy drop when predicting on images of darker skin tones (p<0.001). GPT-4V accurately identified Fitzpatrick skin tones for 56.5% of images. For the multiple-choice styled USMLE image-based test questions, GPT-4V had an accuracy of 59%. Our findings demonstrate that the current version of GPT-4V is limited in its diagnostic abilities across multiple image-based medical specialties. Future work should be done to explore LMM's sensitivity to prompting as well as hybrid models that can combine LMM's capabilities with other robust models. ### Competing Interest Statement R.D. has served as an advisor to MDAlgorithms and Revea and received consulting fees from Pfizer, L'Oreal, Frazier Healthcare Partners, and DWA, and research funding from UCB. All other authors declare no competing interests. ### Funding Statement This study did not receive any funding. ### Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes The data used in this study is available in the supplementary material.