Is ChatGPT an Evidence-based Doctor?

To assess chat generative pre-trained transformer's (ChatGPT) performance on the American Urological Association Self-Assessment Study Program (AUA SASP) and stratify performance by question stem complexity.Questions from the 2021-2022 AUA SASP program were administered to ChatGPT version 3 (ChatGPT-3). Questions were administered to the model utilizing a standardized prompt. The answer choice selected by ChatGPT was then used to answer the question stem in the AUA SASP program. ChatGPT was then prompted to assign a question stem order (first, second, third) to each question. The percentage of correctly answered questions was determined for each order level. All responses provided by ChatGPT were qualitatively assessed for appropriate rationale.A total of 268 questions were administered to ChatGPT. ChatGPT performed better on 2021 compared to the 2022 AUA SASP question set, answering 42.3% versus 30.0% of questions correctly (P < .05). Hundred percent of answer explanations provided appropriate, relevant rationale regardless of whether the answer was correct. Further stratification included assessment by question order level. ChatGPT performed progressively better on the 2021 question set with decreasing order levels, with first-order questions reaching 53.8% (n = 14). However, differences in proportions did not reach statistical significance (P > .05).ChatGPT answered many high-level questions correctly and provided a reasonable rationale for each answer choice. While ChatGPT was unable to answer numerous first-order questions, future language processing model learning may lead to the optimization of its fund of knowledge. This may lead to the utilization of artificial intelligence like ChatGPT as an educational tool for urology trainees and professors.