Thyroid nodule recognition using a joint convolutional neural network with information fusion of ultrasound images and radiofrequency data

Objective To develop a deep learning–based method with information fusion of US images and RF signals for better classification of thyroid nodules (TNs). Methods One hundred sixty-three pairs of US images and RF signals of TNs from a cohort of adult patients were used for analysis. We developed an information fusion–based joint convolutional neural network (IF-JCNN) for the differential diagnosis of malignant and benign TNs. The IF-JCNN contains two branched CNNs for deep feature extraction: one for US images and the other one for RF signals. The extracted features are fused at the backend of IF-JCNN for TN classification. Results Across 5-fold cross-validation, the accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC) obtained by using the IF-JCNN with both US images and RF signals as inputs for TN classification were respectively 0.896 (95% CI 0.838–0.938), 0.885 (95% CI 0.804–0.941), 0.910 (95% CI 0.815–0.966), and 0.956 (95% CI 0.926–0.987), which were better than those obtained by using only US images: 0.822 (0.755–0.878; p = 0.0044), 0.792 (0.679–0.868, p = 0.0091), 0.866 (0.760–0.937, p = 0.197), and 0.901 (0.855–0.948, p = .0398), or RF signals: 0.767 (0.694–0.829, p < 0.001), 0.781 (0.685–0.859, p = 0.0037), 0.746 (0.625–0.845, p < 0.001), 0.845 (0.786–0.903, p < 0.001). Conclusions The proposed IF-JCNN model filled the gap of just using US images in CNNs to characterize TNs, and it may serve as a promising tool for assisting the diagnosis of thyroid cancer. Key Points • Raw radiofrequency signals before ultrasound imaging of thyroid nodules provide useful information that is not carried by ultrasound images. • The information carried by raw radiofrequency signals and ultrasound images for thyroid nodules is complementary. • The performance of deep convolutional neural network for diagnosing thyroid nodules can be significantly improved by fusing US images and RF signals in the model as compared with just using US images.