The success of large-scale pretrained models has established fine-tuning as a standard method for achieving significant improvements in downstream tasks. However, fine-tuning the entire parameter set of a pretrained model is costly. Parameter-efficient transfer learning (PETL) has recently emerged as a cost-effective alternative for adapting pretrained models to downstream tasks. Despite its advantages, the increasing model size and input resolution present challenges for PETL, as the training memory consumption is not reduced as effectively as the parameter usage. In this article, we introduce fine-grained prompt tuning plus (FPT+), a PETL method designed for high-resolution medical image classification, which significantly reduces the training memory consumption compared to other PETL methods. FPT+ performs transfer learning by training a lightweight side network and accessing pretrained knowledge from a large pretrained model (LPM) through fine-grained prompts and fusion modules. Specifically, we freeze the LPM of interest and construct a learnable lightweight side network. The frozen LPM processes high-resolution images to extract fine-grained features, while the side network employs corresponding downsampled low-resolution images to minimize memory usage. To enable the side network to leverage pretrained knowledge, we propose fine-grained prompts and fusion modules, which collaborate to summarize information through the LPM’s intermediate activations. We evaluate FPT+ on eight medical image datasets of varying sizes, modalities, and complexities. Experimental results demonstrate that FPT+ outperforms other PETL methods, using only 1.03% of the learnable parameters and 3.18% of the memory required for fine-tuning an entire ViT-B model. Our code is available https://github.com/YijinHuang/FPT.

In the pursuit of advancing medical diagnosis, automatic segmentation of medical images is crucial, particularly in extending medical expertise to under-resourced regions. However, collecting and annotating medical data for deep learning frameworks are both time-consuming and expensive. Few-shot learning, which leverages limited labeled data to learn new tasks, has been widely applied to medical image segmentation, offering significant advancements. Nonetheless, these methods often rely on extensive unlabeled data to acquire prior medical knowledge. We introduce SnapSeg, a novel few-shot segmentation framework that stands out by requiring only a minimal set of labeled images to directly tackle new segmentation tasks, thus bypassing the need for a traditional training phase. Utilizing either a single or a few labeled examples, SnapSeg extracts multi-level features from the Segment Anything Model (SAM)'s image encoder and incorporates a relative anchor algorithm for precise spatial assessment. Our method demonstrates state-of-the-art performance on the widely-used Abd-CT dataset in medical image segmentation.

In medical contexts, the imbalanced data distribution in long-tailed datasets, due to scarce labels for rare diseases, greatly impairs the diagnostic accuracy of deep learning models. Recent multimodal text-image supervised foundation models offer new solutions to data scarcity through effective representation learning. However, their limited medical-specific pretraining hinders their performance in medical image classification relative to natural images. To address this issue, we propose a novel Text-guided Foundation model Adaptation for Long-Tailed medical image classification (TFA-LT). We adopt a two-stage training strategy, integrating representations from the foundation model using just two linear adapters and a single ensembler for balanced outcomes. Experimental results on two long-tailed medical image datasets validate the simplicity, lightweight and efficiency of our approach: requiring only 6.1% GPU memory usage of the current best-performing algorithm, our method achieves an accuracy improvement of up to 27.1%, highlighting the substantial |potential of foundation model adaptation in this area.

Medical image datasets often exhibit long-tailed distributions due to the inherent challenges in medical data collection and annotation. In long-tailed contexts, some common disease categories account for most of the data, while only a few samples are available in the rare disease categories, resulting in poor performance of deep learning methods. To address this issue, previous approaches have employed class resampling or re-weighting techniques, which often encounter challenges such as overfitting to tail classes or difficulties in optimization during training. In this work, we propose a novel approach, namely Saliency-guided and Patch-based Mixup (SPMix) for long-tailed skin cancer image classification. Specifically, given a tail-class image and a head-class image, we generate a new tail-class image by mixing them under the guidance of saliency mapping, which allows for preserving and augmenting the discriminative features of the tail classes without any interference of the head-class features. Extensive experiments are conducted on the ISIC2018 dataset, demonstrating the superiority of SPMix over existing state-of-the-art methods. The source code is available at https://github.com/Yancy10-1/SPMix.

Unsupervised anomaly detection methods aim to reduce the cost of manually annotating abnormal medical image datasets. However, since they are not trained on massive abnormal images, their discriminative capability may be low. In this paper, we present AugPaste, a novel one-shot anomaly detection framework for detecting diabetic retinopathy (DR) from fundus images. AugPaste utilizes true anomalies from a single annotated DR sample to synthesize a large amount of artificial DR fundus images. The framework begins with constructing a DR lesion bank through augmentation of randomly selected DR lesion patches. Synthesized DR samples are then generated by pasting lesion patches selected from the lesion bank into normal images using various prior knowledge-guided strategies. We finally train a classification network on the synthetic abnormal images along with true normal images for anomaly detection. Our tests on four public fundus image datasets show that AugPaste outperforms leading unsupervised and few-shot methods and rivals fully-supervised methods. The source code is available at https://github.com/Aidanvk/AugPaste.

Self-supervised Learning (SSL) has been widely applied to learn image representations through exploiting unlabeled images. However, it has not been fully explored in the medical image analysis field. In this work, Saliency-guided Self-Supervised image Transformer (SSiT) is proposed for Diabetic Retinopathy (DR) grading from fundus images. We novelly introduce saliency maps into SSL, with a goal of guiding self-supervised pre-training with domain-specific prior knowledge. Specifically, two saliency-guided learning tasks are employed in SSiT: (1) Saliency-guided contrastive learning is conducted based on the momentum contrast, wherein fundus images' saliency maps are utilized to remove trivial patches from the input sequences of the momentum-updated key encoder. Thus, the key encoder is constrained to provide target representations focusing on salient regions, guiding the query encoder to capture salient features. (2) The query encoder is trained to predict the saliency segmentation, encouraging the preservation of fine-grained information in the learned representations. To assess our proposed method, four publicly-accessible fundus image datasets are adopted. One dataset is employed for pre-training, while the three others are used to evaluate the pre-trained models' performance on downstream DR grading. The proposed SSiT significantly outperforms other representative state-of-the-art SSL methods on all downstream datasets and under various evaluation settings. For example, SSiT achieves a Kappa score of 81.88% on the DDR dataset under fine-tuning evaluation, outperforming all other ViT-based SSL methods by at least 9.48%.

Medical image classification is a challenging task, particularly when dealing with long-tailed datasets where rare diseases are underrepresented. The imbalanced class distribution in such datasets poses significant challenges in accurately classifying minority classes. Existing methods for alleviating the long-tail problem in medical image classification suffer from limitations such as noise introduction, loss of crucial information, and the need for manual tuning and additional computational resources. In this study, we propose a novel framework called Multiple Expert Decoupled Knowledge Distillation (MEDKD) to tackle the imbalanced class distribution in medical image classification. The knowledge distillation of multiple teacher models can significantly alleviate the class imbalance by partitioning the dataset into several subsets. However, current frameworks of this kind have not yet explored the integration of more advanced distillation methods. Our framework incorporating TCKD and NCKD concepts to improve classification performance. Through comprehensive experiments on publicly available datasets, we evaluate the performance of MEDKD and compare it with state-of-the-art methods. Our results demonstrate remarkable accuracy improvements achieved by the proposed method, highlighting its effectiveness in alleviating the challenges of medical image classification with long-tailed datasets.

Medical image quality assessment (MIQA) is a vital prerequisite in various medical image analysis applications. Most existing MIQA algorithms are fully supervised that request a large amount of annotated data. However, annotating medical images is time-consuming and labor-intensive. In this paper, we propose an unsupervised anomaly-aware framework with test-time clustering for optical coherence tomography angiography (OCTA) image quality assessment in a setting wherein only a set of high-quality samples are accessible in the training phase. Specifically, a feature-embedding-based low-quality representation module is proposed to quantify the quality of OCTA images and then to discriminate between outstanding quality and non-outstanding quality. Within the non-outstanding quality class, to further distinguish gradable images from ungradable ones, we perform dimension reduction and clustering of multi-scale image features extracted by the trained OCTA quality representation network. Extensive experiments are conducted on one publicly accessible dataset sOCTA-3*3-10k, with superiority of our proposed framework being successfully established.

Diabetic retinopathy (DR) is one of the leading causes of blindness. Early DR screening may significantly reduce the risk of vision impairments. Traditional DR diagnosis relies on a professional ophthalmologist to examine the fundus image of a potential patient. However, in remote and underdeveloped areas, medical resources are relatively limited, hindering professional and timely DR diagnosis. In such context, there is an urgent need to develop convenient and fast DR screening methods to provide effective intervention plans. In this work, we implemented and validated a real-time DR diagnosis system on field-programmable gate array (FPGA) board. We first trained a DR grading CNN of interest on GPUs, and then deployed it on FPGA after quantization. Two CNN architectures, ResNet50 and MobileNetV2, are investigated. With 16-bit quantization, ResNet50 and MobileNetV2 respectively achieve 0.828 and 0.753 Kappa scores on the publicly-accessible EyePACS test set. Improvements in average processing time have also been observed: 16.92ms over CPU and 0.21ms over GPU.

Deep learning-based methods have been widely used for medical image classification. However, in clinical practice, rare diseases are usually underrepresented with limited labeled data, which result in long-tailed medical datasets and significantly degrade the performance of deep classification networks. Previous strategies employ re-sampling or re-weighting techniques to alleviate this issue by increasing the influence of underrepresented classes and reducing the influence of overrepresented ones. Still, poor performance may occur due to overfitting of the tail classes. Further, Mixup is employed to introduce additional information into model training. Despite considerable improvements, the significant noise in medical images means that random batch mixing may introduce ambiguity into training, thereby impair the performance. This observation motivates us to develop a fine-grained mixing approach. In this paper we present Curriculum of Class-wise Mixup (CCMix), a novel method for addressing the challenge of long-tailed distributions. CCMix leverages a novel curriculum that takes into account both the degree of mixing and the class-wise performance to identify the ideal Mixup proportions of different classes. Our method's simplicity enables its effortless integration with existing long-tailed recognition techniques. Comprehensive experiments on two long-tailed medical image classification datasets demonstrate that our method, requiring no modifications to the framework structure or algorithmic details, achieves state-of-the-art results across diverse long-tailed classification benchmarks. The source code is available at https://github.com/sirileeee/CCMix .

The quality of a fundus image can be compromised by numerous factors, many of which are challenging to be appropriately and mathematically modeled. In this paper, we introduce a novel diffusion model based framework, named Learning Enhancement from Degradation (LED), for enhancing fundus images. Specifically, we first adopt a data-driven degradation framework to learn degradation mappings from unpaired high-quality to low-quality images. We then apply a conditional diffusion model to learn the inverse enhancement process in a paired manner. The proposed LED is able to output enhancement results that maintain clinically important features with better clarity. Moreover, in the inference phase, LED can be easily and effectively integrated with any existing fundus image enhancement framework. We evaluate the proposed LED on several downstream tasks with respect to various clinically-relevant metrics, successfully demonstrating its superiority over existing state-of-the-art methods both quantitatively and qualitatively. The source code is available at https://github.com/QtacierP/LED.

Contrastive learning based vision-language joint pre-training has emerged as a successful representation learning strategy. In this paper, we present a prototype representation learning framework incorporating both global and local alignment between medical images and reports. In contrast to standard global multi-modality alignment methods, we employ a local alignment module for fine-grained representation. Furthermore, a cross-modality conditional reconstruction module is designed to interchange information across modalities in the training phase by reconstructing masked images and reports. For reconstructing long reports, a sentence-wise prototype memory bank is constructed, enabling the network to focus on low-level localized visual and high-level clinical linguistic features. Additionally, a non-auto-regressive generation paradigm is proposed for reconstructing non-sequential reports. Experimental results on five downstream tasks, including supervised classification, zero-shot classification, image-to-text retrieval, semantic segmentation, and object detection, show the proposed method outperforms other state-of-the-art methods across multiple datasets and under different dataset size settings. The code is available at https://github.com/QtacierP/PRIOR.

Computer-aided diagnosis (CAD) of prostate cancer (PCa) using multi-parametric magnetic resonance imaging (mp-MRI) has recently gained great research interest. In this work, a fully automatic CAD pipeline of PCa using mp-MRI data is presented. In order to fully explore the mp-MRI data, we systematically investigate three multi-modal medical image fusion strategies in convolutional neural networks, namely input-level fusion, feature-level fusion, and decision-level fusion. Extensive experiments are conducted on two datasets with different PCa-related diagnostic tasks. We identify a pipeline that works relatively the best for both diagnostic tasks, two important components of which are stacking three adjacent slices as the input and performing decision-level fusion with specific loss weights. Clinical relevance- This work provides a practical method for automated diagnosis of PCa based on multi-parametric MRI

Convolutional neural networks have been widely applied to medical image segmentation and have achieved considerable performance. However, the performance may be significantly affected by the domain gap between training data (source domain) and testing data (target domain). To address this issue, we propose a data manipulation based domain generalization method, called Automated Augmentation for Domain Generalization (AADG). Our AADG framework can effectively sample data augmentation policies that generate novel domains and diversify the training set from an appropriate search space. Specifically, we introduce a novel proxy task maximizing the diversity among multiple augmented novel domains as measured by the Sinkhorn distance in a unit sphere space, making automated augmentation tractable. Adversarial training and deep reinforcement learning are employed to efficiently search the objectives. Quantitative and qualitative experiments on 11 publicly-accessible fundus image datasets (four for retinal vessel segmentation, four for optic disc and cup (OD/OC) segmentation and three for retinal lesion segmentation) are comprehensively performed. Two OCTA datasets for retinal vasculature segmentation are further involved to validate cross-modality generalization. Our proposed AADG exhibits state-of-the-art generalization performance and outperforms existing approaches by considerable margins on retinal vessel, OD/OC and lesion segmentation tasks. The learned policies are empirically validated to be model-agnostic and can transfer well to other models. The source code is available at https://github.com/CRazorback/AADG.

Although deep learning-based diabetic retinopathy (DR) classification methods typically benefit from well-designed architectures of convolutional neural networks, the training setting also has a non-negligible impact on prediction performance. The training setting includes various interdependent components, such as an objective function, a data sampling strategy, and a data augmentation approach. To identify the key components in a standard deep learning framework (ResNet-50) for DR grading, we systematically analyze the impact of several major components. Extensive experiments are conducted on a publicly available dataset EyePACS. We demonstrate that (1) the DR grading framework is sensitive to input resolution, objective function, and composition of data augmentation; (2) using mean square error as the loss function can effectively improve the performance with respect to a task-specific evaluation metric, namely the quadratically weighted Kappa; (3) utilizing eye pairs boosts the performance of DR grading and; (4) using data resampling to address the problem of imbalanced data distribution in EyePACS hurts the performance. Based on these observations and an optimal combination of the investigated components, our framework, without any specialized network design, achieves a state-of-the-art result (0.8631 for Kappa) on the EyePACS test set (a total of 42,670 fundus images) with only image-level labels. We also examine the proposed training practices on other fundus datasets and other network architectures to evaluate their generalizability. Our codes and pre-trained model are available online.

Due to the high cost of manually annotating medical images, especially for large-scale datasets, anomaly detection has been explored through training models with only normal data. Lacking prior knowledge of true anomalies is the main reason for the limited application of previous anomaly detection methods, especially in the medical image analysis realm. In this work, we propose a one-shot anomaly detection framework, namely AugPaste, that utilizes true anomalies from a single annotated sample and synthesizes artificial anomalous samples for anomaly detection. First, a lesion bank is constructed by applying augmentation to randomly selected lesion patches. Then, MixUp is adopted to paste patches from the lesion bank at random positions in normal images to synthesize anomalous samples for training. Finally, a classification network is trained using the synthetic abnormal samples and the true normal data. Extensive experiments are conducted on two publicly-available medical image datasets with different types of abnormalities. On both datasets, our proposed AugPaste largely outperforms several state-ofthe-art unsupervised and semi-supervised anomaly detection methods, and is on a par with the fully-supervised counterpart. To note, AugPaste is even better than the fully-supervised method in detecting early-stage diabetic retinopathy.

Anomalous data are usually rare in the field of medical imaging, in contrast to normal (healthy) data that account for the vast majority of the real-world medical image data, leading to challenges of developing image-based disease detection algorithms. In this work, we propose an unsupervised anomaly detection framework for diabetic retinopathy (DR) identification from fundus images, named Lesion2Void. Lesion2Void is capable of identifying anomalies in fundus images by only leveraging normal data without any additional annotation during training. We first randomly mask out multiple patches in normal fundus images. Then, a convolutional neural network is trained to reconstruct the corresponding complete images. We make a simple assumption that in a fundus image, lesion patches, if present, are independent of each other and are also independent of their neighboring pixels, whereas normal patches can be predicted based on the information from the neighborhood. Therefore, in the testing phase, an image can be identified as normal or abnormal by measuring the reconstruction errors of the erased patches. Extensive experiments are conducted on the publicly accessible dataset EyeQ, demonstrating the superiority of our proposed framework for DR-related anomaly detection in fundus images.

Fundus image quality is crucial for screening various ophthalmic diseases. In this paper, we proposed and validated a novel fundus image enhancement method, named importance-guided semi-supervised contrastive constraining (I-SECRET). Specifically, our semi-supervised framework consists of an unsupervised component, a supervised component, and an importance estimation component. The unsupervised part makes use of a large publicly-available dataset of unpaired high-quality and low-quality images via contrastive constraining, whereas the supervised part utilizes paired images through degrading pre-selected high-quality images. The importance estimation provides a pixel-wise importance map to guide both unsupervised and supervised learning. Extensive experiments on both authentic and synthetic data identify the superiority of our proposed method over existing state-of-the-art ones, both quantitatively and qualitatively.

Fundus images of poor quality may seriously influence clinic judgments. Existing fundus image quality enhancement (FIQE) approaches mainly make use of general image features but no prior domain knowledge. In this paper, we proposed and validated an efficient FIQE method with a prior constraint, named Efficient Prior Contrastive unpaired Generative Adversarial Network (EPC-GAN). Inspired by the contrastive unpaired translation framework, we emphasized local features during the enhancing process via contrastive patchwise samples. Moreover, to utilize high-level features in the fundus domain (such as vessels, optic disc/cup, and even lesions), we designed a fundus prior loss to avoid information modification and over-enhancement. Besides, we presented an efficient network architecture to overcome the high consumption in terms of both time and GPU-memory. Through both qualitative and quantitative experiments on a public dataset EyeQ, we demonstrated the superior performance of our proposed method.

Manually annotating medical images is extremely expensive, especially for large-scale datasets. Self-supervised contrastive learning has been explored to learn feature representations from unlabeled images. However, unlike natural images, the application of contrastive learning to medical images is relatively limited. In this work, we propose a self-supervised framework, namely lesion-based contrastive learning for automated diabetic retinopathy (DR) grading. Instead of taking entire images as the input in the common contrastive learning scheme, lesion patches are employed to encourage the feature extractor to learn representations that are highly discriminative for DR grading. We also investigate different data augmentation operations in defining our contrastive prediction task. Extensive experiments are conducted on the publicly-accessible dataset EyePACS, demonstrating that our proposed framework performs outstandingly on DR grading in terms of both linear evaluation and transfer capacity evaluation.