Evaluating the Potential of LLMs and ChatGPT on Medical Diagnosis and Treatment.

We evaluate the validity, accuracy, and usefulness of ChatGPT-returned medical diagnosis of lung disease based on symptoms described by a human. Specifically, Tuberculosis and its symptoms are selected as the test case and our evaluation follows the directions of (i) medical validity and accuracy of the returned diagnosis in terms of both context and references, (ii) its usefulness to both doctors and patients and (iii) the economic value added to the healthcare system. It is shown that ChatGPT performs well in diagnosing Tuberculosis, but its performance improves when supervised by a human medical expert. In the interest of adding reproducibility and comparability, we propose a novel general evaluation procedure for the medical domain, to be followed when interacting with Large Language Models. This procedure integrates the various steps employed in our evaluation process and encompasses the review indices utilized for quantifying the outcome.