Artificial Intelligence in Medicine: A Systematic Review of Guidelines on Reporting and Interpreting Studies

Abstract Background Reporting guidelines developed for medical artificial intelligence (AI) studies are structured tools that address general and/or AI-specific methodological and reporting issues. We aimed to systematically review published medical AI reporting guidelines and checklists and evaluate aspects that can support the choice of the tool in a particular research context. Methods We searched PubMed, Scopus, and Web of Science until February 2023. Citations and Google were searched in addition. We included peer reviewed articles of reporting guidelines or checklists applicable for medical AI research. Screening, article selection and data extraction was performed in duplicate. We extracted publication details, the guidelines’ aims, target audiences, development process, focus area, structure, number of items and recorded the number of Google Scholar citations as a proxy to usage. Results From 821 records, and additional sources, 24 guidelines were included (4 narrative guidelines, 7 general reporting checklists, 4 study design specific checklists, 9 clinical area specific checklists). 13 studies reported the guideline development methods, 10 guidelines were registered in the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network. In 224 sections, the guidelines contained 704 items in total. The number of items per checklist varied between 10 and 66. Nine guidelines broadly followed the Introduction, Methods, Results, and Discussion (IMRAD) structure, 12 the machine learning pipeline method (i.e., sequential steps from data processing to model training and evaluation) and 3 had other structure. Conclusions Currently there is no consensus about the structure and format about AI reporting guidelines. The guidelines’ structure and level of detail varied significantly which makes difficult for researchers to follow how detailed and standardized a medical AI study report should be. The robustness of development process and support from the literature suggests CONSORT-AI as the most established tool. Such AI extensions of clinical study guidelines may not cover all the application fields of AI in medicine. In certain research contexts, an established checklist for main study types, and a general AI-based checklist may be used in parallel to provide most useful guidance in designing, writing and interpreting medical AI studies.