Convolutional neural networks for the differentiation between benign and malignant renal tumors with a multicenter international computed tomography dataset

Objectives To use convolutional neural networks (CNNs) for the differentiation between benign and malignant renal tumors using contrast-enhanced CT images of a multi-institutional, multi-vendor, and multicenter CT dataset. Methods A total of 264 histologically confirmed renal tumors were included, from US and Swedish centers. Images were augmented and divided randomly 70%:30% for algorithm training and testing. Three CNNs (InceptionV3, Inception-ResNetV2, VGG-16) were pretrained with transfer learning and fine-tuned with our dataset to distinguish between malignant and benign tumors. The ensemble consensus decision of the three networks was also recorded. Performance of each network was assessed with receiver operating characteristics (ROC) curves and their area under the curve (AUC-ROC). Saliency maps were created to demonstrate the attention of the highest performing CNN. Results Inception-ResNetV2 achieved the highest AUC of 0.918 (95% CI 0.873–0.963), whereas VGG-16 achieved an AUC of 0.813 (95% CI 0.752–0.874). InceptionV3 and ensemble achieved the same performance with an AUC of 0.894 (95% CI 0.844–0.943). Saliency maps indicated that Inception-ResNetV2 decisions are based on the characteristics of the tumor while in most tumors considering the characteristics of the interface between the tumor and the surrounding renal parenchyma. Conclusion Deep learning based on a diverse multicenter international dataset can enable accurate differentiation between benign and malignant renal tumors. Critical relevance statement Convolutional neural networks trained on a diverse CT dataset can accurately differentiate between benign and malignant renal tumors. Key points • Differentiation between benign and malignant tumors based on CT is extremely challenging. • Inception-ResNetV2 trained on a diverse dataset achieved excellent differentiation between tumor types. • Deep learning can be used to distinguish between benign and malignant renal tumors. Graphical Abstract