A structure-fusion network for medical image classification

The Convolutional Neural Networks and Transformer cannot provide satisfactory performance in medical image classification due to insufficient data, high resolution, and a lot of redundancy. To achieve better performance, this paper proposes a structure-fusion network that combines the architecture of convolution and transformer. To reduce the computational overhead incurred by the transformer structure, we optimize it by aggregating adjacent features. We further modify the Multilayer Perceptron using convolution to increase the network capacity. The network is verified on the grading of the Anterior Cruciate Ligament from knee MRI and the screening of pneumonia and COVID-19 from Chest X-ray. Compared with current advanced methods, the proposed network not only reduces FLOPs but also achieves improvement on AUC and F1score.