A Model-Agnostic Framework for Universal Anomaly Detection of Multi-organ and Multi-modal Images

The recent success of deep learning relies heavily on the large amount of labeled data. However, acquiring manually annotated symptomatic medical images is notoriously time-consuming and laborious, especially for rare or new diseases. In contrast, normal images from symptom-free healthy subjects without the need of manual annotation are much easier to acquire. In this regard, deep learning based anomaly detection approaches using only normal imagesare actively studied, achieving significantly better performance than conventional methods. Nevertheless, the previousworks committed to develop a specific network for each organ and modality separately, ignoring the intrinsic similarity among images within medical field. In this paper, we propose a model-agnostic framework to detect the abnormalities of various organs and modalitieswith a single network. By imposing organ and modality classification constraints along with center constraint on the disentangled latent representation, the proposed framework not only improves the generalization ability of thenetwork towards thesimultaneous detectionof anomalousimageswith various organs and modalities, but also boosts the performance on each single organ and modality. Extensive experiments with four different baseline models on three public datasets demonstrate the superiority of the proposed framework as well as the effectiveness of each component.