Weight Averaging (WA) has emerged as a powerful technique for enhancing generalization by promoting convergence to a flat loss landscape, which correlates with stronger out-of-distribution performance. However, applying WA directly to multi-modal domain generalization (MMDG) is challenging: differences in optimization speed across modalities lead WA to overfit to faster-converging ones in early stages, suppressing the contribution of slower yet complementary modalities, thereby hindering effective modality fusion and skewing the loss surface toward sharper, less generalizable minima. To address this issue, we propose MBCD, a unified collaborative distillation framework that retains WA's flatness-inducing advantages while overcoming its shortcomings in multi-modal contexts. MBCD begins with adaptive modality dropout in the student model to curb early-stage bias toward dominant modalities. A gradient consistency constraint then aligns learning signals between uni-modal branches and the fused representation, encouraging coordinated and smoother optimization. Finally, a WA-based teacher conducts cross-modal distillation by transferring fused knowledge to each uni-modal branch, which strengthens cross-modal interactions and steer convergence toward flatter solutions. Extensive experiments on MMDG benchmarks show that MBCD consistently outperforms existing methods, achieving superior accuracy and robustness across diverse unseen domains.

The joint interpretation of multi-modal and multi-field fundus images is critical for retinopathy prevention, as different fields can show the complete 3D eyeball field and different modalities can provide complementary lesion areas. Compared with single images, the sequence relationships in multi-modal and multi-field fundus images contain long-range dependencies in lesion features. By modeling the global contextual dependencies in these sequences, lesion areas can be more comprehensively mined, and modality-specific lesions can be detected. To learn the long-range contexts and fuse complementary multi-scale lesion features between different fundus modalities, we design a multi-modal fundus image fusion method based on multi-scale cross-attention, which solves the static receptive field problem in previous multi-modal medical fusion methods based on attention. To capture multi-field relative positional relationships between different fields and fuse comprehensive lesion features between different fields, we design a multi-field fundus image fusion method based on shifted window self-attention, which also solves the computational complexity of the multi-field fundus fusion method based on self-attention is quadratic to the size and number of multi-field fundus images. Finally, we design a multi-task retinopathy diagnosis framework to help ophthalmologists reduce workload and improve diagnostic accuracy by combining the proposed two fusion methods. The experimental results of retinopathy classification and report generation tasks indicate our method’s potential to improve the efficiency and reliability of retinopathy diagnosis in clinical practice, achieving a classification accuracy of 82.53% and a report generation BlEU-1 of 0.543.

Weakly supervised semantic segmentation (WSSS) suffers from an inherent mismatch between coarse image-level annotations and dense pixel-level predictions. To bridge this gap, existing methods primarily focus on generating refined class activation maps (CAM) as pseudo-labels. However, we argue that this focus is insufficient as it overlooks a critical component: the segmentation decoder. The decoder is typically trained through superficial alignment of predictions with pseudo-labels in the logit space. Given the noisy nature of such labels, this naive supervision leads to error accumulation and limits performance. To address this, we propose an Uncertainty-Guided Reliable Learning (UGRL) framework that exerts dual control to reshape the learning process, achieving robust supervision that escapes the CAM shadow. The cornerstone of UGRL is a prototype-driven uncertainty modeling module that estimates the reliability of class-wise supervision. The modeled uncertainty enables two synergistic control mechanisms. First, it adaptively modulates classification and segmentation losses, encouraging the model to learn from more trustworthy signals. Second, it guides the structuring of the decoder’s feature space. Rather than relying solely on superficial alignment, UGRL enforces deeper representation alignment by applying contrastive learning on reliable pixels. This enables rich semantic transfer to fine-grained segmentation details. Extensive experiments on PASCAL VOC and MS COCO demonstrate that our method surpasses other state-of-the-art WSSS methods.

Multi-modal and multi-view ophthalmic images are critical tools in clinical eye examinations as they can provide biomarkers and a wide field for the retinal disease diagnosis. However, analyzing ophthalmic images remains a challenge due to the long-range dependencies consisting of correspondence between different modalities and complementarity between different perspectives. Inspired by these encouraging characteristics of the Mamba framework for balancing between global contextual dependencies modeling and linear time complexity, we propose a novel ophthalmic image analysis framework based on Mamba to learn global contextual representations and fuse information between different modalities and views. Specifically, we propose a corresponding selective scanning strategy to model long-range dependencies between the correlated regions of different modalities. Furthermore, to mine the complementary long-range dependency information among different fundus perspectives, we propose a multi-view Mamba-based scanning strategy. Extensive experiments have been conducted on retinopathy diagnosis tasks to evaluate the effectiveness of our model for diagnosing retinopathy.

The proliferation of harmful memes on online media poses significant risks to public health and stability. Existing detection methods heavily rely on large-scale labeled data for training, which necessitates substantial manual annotation efforts and limits their adaptability to the continually evolving nature of harmful content. To address these challenges, we present ALARM, the first lAbeL-free hARmful Meme detection framework powered by Large Multimodal Model (LMM) agent self-improvement. The core innovation of ALARM lies in exploiting the expressive information from "shallow" memes to iteratively enhance its ability to tackle more complex and subtle ones. ALARM consists of a novel Confidence-based Explicit Meme Identification mechanism that isolates the explicit memes from the original dataset and assigns them pseudo-labels. Besides, a new Pairwise Learning Guided Agent Self-Improvement paradigm is introduced, where the explicit memes are reorganized into contrastive pairs (positive vs. negative) to refine a learner LMM agent. This agent autonomously derives high-level detection cues from these pairs, which in turn empower the agent itself to handle complex and challenging memes effectively. Experiments on three diverse datasets demonstrate the superior performance and strong adaptability of ALARM to newly evolved memes. Notably, our method even outperforms label-driven methods. These results highlight the potential of label-free frameworks as a scalable and promising solution for adapting to novel forms and topics of harmful memes in dynamic online environments.

Fundus images are widely used in early retinopathy examination to prevent visual impairment caused by retinopathy. The retinopathy examination process based on fundus images can be mainly summarized in three steps: (1) ophthalmologists obtain comprehensive fundus information by jointly analyzing multi-view fundus images; (2) ophthalmologists obtain complementary lesion information by contrastingly analyzing multi-modal fundus images; (3) ophthalmologists diagnose retinopathy categories and write specialized fundus reports. To simulate the clinical fundus image examination process, we introduce an efficient multi-view and multi-modal fundus image joint ancillary diagnosis framework that can simultaneously accept fundus images of different views and modalities for pathology classification and symptom report generation tasks. In our framework, we propose jointly employing self-attention in intra-view local and inter-view sparse global windows to extract comprehensive fundus information among different views. We propose a multi-modal fusion transformer via shunted multi-scale cross-attention to model lesions of various scales by splitting attention granularity at query and queried modalities to fuse complementary lesion information among different modalities. The experimental results of retinopathy classification and report generation tasks indicate that our proposed method is superior to other benchmarking methods, achieving a classification accuracy of 83.96% and a report generation CIDEr of 0.934.

Image clustering is a challenging task in computer vision, with performance heavily dependent on the quality of feature representations due to the inherent complexity of images. However, current image clustering methods overlook the underlying semantic information during representation learning, leading to low-quality feature representations. Moreover, the absence of ground-truth labels amplifies the detrimental effects of unreliable data on semantic guidance, steering the model towards incorrect learning directions. In this work, we propose a novel deep image clustering method named Uncertainty-aware Correspondence Distillation (UCD) to address these issues. Specifically, we introduce the concept of representation correspondence to establish cross-level connections between instances and semantics, which is further employed as a distillation target to improve the network’s feature learning by complementing semantic information. To mitigate unnecessary similarity penalties arising from unreliable data, we develop robust dynamic weights for semantic guidance by modeling the uncertainty of image semantics. Extensive experiments on five benchmark datasets demonstrate the superiority of the proposed method. The code is available at https://github.com/YL616/UCD.

Deep clustering aims to group unlabeled data into meaningful clusters by learning discriminative feature representations. However, ambiguous features often lead to noisy representations and inconsistent semantics, limiting improvements in clustering performance. To address this issue, we propose an Uncertainty-Aware Contrastive Learning (UACL) method for deep clustering, which achieves robustness by adaptively restricting the learning of ambiguous features. Specifically, we model pairwise similarity evidence via subjective logic theory, formulating co-cluster probabilities as a Dirichlet distribution to quantify epistemic uncertainty from feature ambiguity. Guided by this uncertainty, we design a dynamic weight-updating strategy that progressively extracts information from potential positives, enhancing the model's ability to learn discriminative representations and semantically consistent clusters. Furthermore, to enforce attribute consistency, we develop an Attribute Distribution Alignment module that aligns similarity and uncertainty. Extensive experiments on five benchmark datasets demonstrate UACL outperforms current state-of-the-art methods, with an improved ACC of 3.5% for CIFAR-100 and 7.0% for ImageNet-Dogs. The source code is available at: https://github.com/YL616/ UACL.

Cross-modal retrieval aims to retrieve relevant content of different modalities by giving a query of another modality. The biggest difficulty is how to bridge the heterogeneous gap between different modalities. The commonly-used methods tend to focus on exploiting individual image-text pair and mining the relations of cross-modality data thereof, but ignore the role of multi-sample correlation. Moreover, more global, structural inter-pair knowledge contained by the training dataset will be under-used. To fully exploit graph-structured semantics and mine the semantic information in the dataset for learning discriminative representations, we propose Weighted Graph-structured Semantics Constraint Network (WGSCN), a unified, graph-based, semantic-constrained learning framework, in which GCN is used to mine comprehensive relation information from cross modality data. Our main inspiration is to design a novel two-branch GCN-based Cross-modal Semantic Encoding (GCSE) module to produce semantic embeddings with the both modality-specific and modality-shared correlation. Moreover, a GAN-based dual learning approach is used to further improve the discriminability and model the joint distribution across different modalities. Our proposed GDL uses semantic embeddings as supervisory signal to make the common representation semantically discriminative while adversarial learning and dual learning are used to make the common representation modality-invariant. Through comparative experiments on five commonly used cross-modal datasets, we have shown the superior retrieval accuracy of our WGSCN.

Modern deep learning has many drawbacks, including a heavy reliance on labeled data. One of the key strategies for solving this problem is few-shot learning (FSL). With just a few labeled samples, FSL seeks to identify previously unknown classes. The majority of works that have been published thus far focus on comparing the features of query samples and support classes, which do not fully utilize the training set's data and do not help sustain performance improvement. In our study, we compute a new attribute distribution similarity between support classes and a query sample of novel classes using attribute information on the training set. We suggest a fresh approach to three phases to accomplish our objective: 1) A attribute provider harnesses the visual features of the training set to construct attributes. 2) Choosing appropriate attributes for novel classes and enriching attributes to determine how similar novel classes and attributes are to one another. 3) To help with classification, attribute distribution similarity is computed for the first time by creating new correlations between the support classes and the query samples, which increases the accuracy of picture classification. Be aware that our solution won't make the initial network settings larger. Experiments on inductive FSL tasks demonstrate the usefulness and practicality of our strategy. Specifically, Our method has achieved the highest performance in the 5-way 1-shot task settings on the tiered-ImageNet and CUB 200–2011 datasets, with impressive results of 73.22% and 82.34% respectively.

IntroductionRecent efforts have been made to apply machine learning and deep learning approaches to the automated classification of schizophrenia using structural magnetic resonance imaging (sMRI) at the individual level. However, these approaches are less accurate on early psychosis (EP) since there are mild structural brain changes at early stage. As cognitive impairments is one main feature in psychosis, in this study we apply a multi-task deep learning framework using sMRI with inclusion of cognitive assessment to facilitate the classification of patients with EP from healthy individuals.MethodUnlike previous studies, we used sMRI as the direct input to perform EP classifications and cognitive estimations. The proposed deep learning model does not require time-consuming volumetric or surface based analysis and can provide additionally cognition predictions. Experiments were conducted on an in-house data set with 77 subjects and a public ABCD HCP-EP data set with 164 subjects.ResultsWe achieved 74.9 ± 4.3% five-fold cross-validated accuracy and an area under the curve of 71.1 ± 4.1% on EP classification with the inclusion of cognitive estimations.DiscussionWe reveal the feasibility of automated cognitive estimation using sMRI by deep learning models, and also demonstrate the implicit adoption of cognitive measures as additional information to facilitate EP classifications from healthy controls.

Automated radiology report generation is gaining popularity as a means to alleviate the workload of radiologists and prevent misdiagnosis and missed diagnoses. By imitating the working patterns of radiologists, previous report generation approaches have achieved remarkable performance. However, these approaches suffer from two significant problems: (1) lack of visual prior: medical observations in radiology images are interdependent and exhibit certain patterns, and lack of such visual prior can result in reduced accuracy in identifying abnormal regions; (2) lack of alignment between images and texts: the absence of annotations and alignments for regions of interest in the radiology images and reports can lead to inconsistent visual and textual features of the abnormal regions generated by the model. To address these issues, we propose a Visual Prior-based Cross-modal Alignment Network for radiology report generation. First, we propose a novel Contrastive Attention that compares input image with normal images to extract difference information, namely visual prior, which helps to identify abnormalities quickly. Then, to facilitate the alignment of images and texts, we propose a Cross-modal Alignment Network that leverages the cross-modal matrix initialized by the features generated by pre-trained models, to compute cross-modal responses for visual and textual features. Finally, a Visual Prior-guided Multi-Head Attention is proposed to incorporate the visual prior into the generation process. The extensive experimental results on two benchmark datasets, IU-Xray and MIMIC-CXR, illustrate that our proposed model outperforms the state-of-the-art models over almost all metrics, achieving BLEU-4 scores of 0.188 and 0.116 and CIDEr scores of 0.409 and 0.240, respectively.

Early eye examination based on fundus images effectively prevents visual impairment caused by retinopathy. The laborious and error-prone process of interpreting fundus images and the lack of ophthalmologists have driven research toward automated retinopathy diagnosis. However, most previous studies have focused on single-modality fundus images, disregarding the integration of information from multiple views, rendering the results unsatisfactory and inconsistent with clinical practice due to the incomprehensive lesion features and incomplete fundus fields. To address this issue, we introduce multi-modality and multi-view fundus images into the automated retinopathy diagnosis pipeline. In contrast to single fundus images, sequential relationships in multi-modality and multi-view fundus images contain essential long-range dependency information, which is vital for retinopathy diagnosis. Inspired by the recent success of transformers for excavating long-range dependencies in sequence data, in this paper, we propose a transformer-based automated retinopathy diagnosis framework for pathology classification and symptom report generation by integrating multi-modality and multi-view fundus images. Specifically, we present two transformer-based networks to construct long-range dependencies in different fundus images. Moreover, we adopt two novel modules to aggregate features of different modalities and views by modeling long-range dependencies among different fundus image sequences. Experiments are conducted on two in-house datasets, in which each subject provides one color fundus photography image and four-view fundus fluorescein angiography images. The experimental results of retinopathy classification and report generation tasks indicate that our proposed method is superior to other benchmarking methods, achieving a classification accuracy of 85.49% and a report generation BlEU-1 of 0.422.

Exploiting relationship among samples in cross-modal data plays a key role in the task of cross modal retrieval, but most of existing methods only extract the correlation from pairwise samples and ignore the relations of unpaired samples. Some graph regularization methods proposed a reasonable paradigm to exploit the correlation from multiple samples. However, limited by the traditional framework, the performance has much room to improve. Moreover, although some existing DNN-based methods achieve excellent performance, the requirement of massive labeled data is also a shortcoming. In this paper, we propose a novel semi-supervised method, named Semi-supervised Constrained Graph Convolutional Network (SCGCN), which adopts graph convolutional network to exploit correlation from batch samples of data with different modalities. For reducing the requirement of labeled data, we design a two stage training procedure: deep supervised learning stage and unsupervised learning stage. In deep supervised learning stage, we integrate two DNN-based semantic encoding networks and a shared classifier into Deep Cross-modal Semantic Encoding (DCSE) module which is trained by supervised learning with labeled data. From DCSE module, we learn a temporary modality-invariant space where the semantic embeddings of samples with different modalities are modality-invariant, and we also learn a classifier which can generate predicted label from the unlabeled data. In unsupervised learning stage, for fully exploiting the correlation from cross-modal data, we design a Constrained Graph Convolutional Network (CGCN) module which utilizes GCN to exploit the correlation and adopts both intra-modal discriminative loss and inter-modal pairwise similar loss to ensure the generated common representation modality-invariant and semantical discriminative. We perform extensive experiments on four conventional datasets and a large scale dataset to demonstrate the effectiveness of proposed approach.

Although deep learning has achieved promising results on medical image classification, the domain shift between training and testing datasets leads to a low prediction accuracy. Domain adaptation is a effective solution. However, due to privacy issues and the lack of annotated data, it’s hard for conventional domain adaptation methods to access source images and labeled target images. To tackle this issue, we propose a novel framework that only requires unlabeled target domain data. This framework has two modules, one is based on class conditional generative adversarial net for source domain generation and another is for classification m odel t raining. S pecifically, the generator can generate target-style data as the pseudo-source data using random noise and a given label to improve the classifier. The increased a ccuracy of the classifier also can guide the generator. Besides, we introduce weight regularization and clustering-based regularization to keep the training process stable and fully explore the discriminative information. We take diabetic retinopathy grade classification as our task and conduct experiments on three datasets which are EysPACS, MESSIDOR and IDRiD. The experimental results show that our method performs well on only unlabeled target data, which proves that it is a general method and can be widely used in the field of medical image classification.

Despite the recent success of deep learning-based models for medical image segmentation and the importance of automated fluorescein leakage segmentation for the diagnosis of advanced diabetic retinopathy, segmentation of fluorescein leakage has been neglected because 1) there are no publicly available databases with sufficient annotations to train segmentation models and 2) supervised models struggle to accurately distinguish between different types of fluorescein leakage and localize leakages at different imaging angles. To tackle these challenges, this work presents FLeak-Seg, a cross-modal dual attention learning method to jointly capture visual and language information, for end-to-end fluorescein leakage segmentation in fundus fluorescein angiography. Specifically, both image and text data are used as input, where visual and linguistic features are captured by a cross-modal attention learning module to compensate for the lack of annotations. A keyword classification module is also employed to identify meaningful expressions related to the type and location of fluorescein leakages to further facilitate the segmentation. Experimental results obtained in an in-house fundus fluorescein angiography database demonstrate the superiority of our method. We show how erroneous segmentation masks can be improved using FLeak-Seg, its advantages in the context of limited samples, and its behavior on segmenting different types of fluorescein leakages.

The accurate approximation of pixel value for preserving image details at a high concentration of noise has led the researchers to improve filters performance. A few image restoration filters are effective at lower density noise. Filters are commonly deployed for cameras, image processing tasks, medical image analysis, guided media data transmission, and real-time machine learning. This article proposes a mathematical model for the exact pixel value estimation at a high noise density for RGB and Gray images. The mathematical model is implemented to fuse statistical reasoning on the optimized mask sizes while preserving image details. Different parameter returns from the median filter, the trimmed median filter, the trimmed mean filter, and mood analysis form a mathematical function. The filter iteratively selects different schemes to calculate pixel values at different noise densities with minimum image information. Different processing masks are analyzed to preserve local data at specific image locations correctly in high density. A robust estimator counts false approximation of pixel values as discontinued, identified, and removed. At the post smoothening process, the filter recovers the misclassification of noise-free pixel and blur effects in the image. The qualitative experiments show satisfactory results in storing the details of the image from any image. The performance of the fusion filter is verified with visual quality and performance analysis matrices such as the image enhancement factor, the similarity indicator and the noise ratio from the peak signal.

The use of deep learning networks for medical image classification is becoming increasingly popular. However, annotated datasets for medical diagnosis are still difficult to obtain due to the Limitations of expertise and expensive consumption. In the absence of datasets, the effectiveness, and robustness of the networks are weak for traditional deep networks for medical diagnosis and problems with these kinds of data issues are quite common. Due to the long-tailed distribution of data and the lack of labeling data for some diseases, traditional deep networks and training techniques can suffer from significant overfitting and poor generalization. We examine the issue of locating the locations of impacted lesions in the absence of data and long-tailed distributions by building on recent developments in contrastive learning. The processing of each image—that is, its division into multiple smaller pictures and comparison between them—improves the ability to identify illnesses. Due to the modest variations between the sick and healthy areas of the disease and the necessity for the network to concentrate on the diseased portions of the disease, contrastive learning allows for the maintenance of intra-class undistorted properties. Our technique can swiftly identify an image’s essential details while masking as much superfluous information a spossible. A s a result of the pre-trained network’s increased sensitivity to image attributes, the network model is better able to generalize to samples and is more resilient when dealing with small sample sizes. Classification models when there are few training samples and a long tail distribution; however, when the training dataset is bigger, our method does not necessarily worse than conventional approaches, because few-shot learning is based on conventional techniques, few-shot learning still trains the classical network with the basis tasks. On the ISIC 2018 and internal ophthalmology datasets, our for the first time tests comparing few-shot learning and traditional classification network approachesd emonstrate that the classical few-shot learning performs better than the traditional deep network approach when training samples are few, and our suggested approach outperforms the classical few-shot learning in task setting on both datasets.

In current state-of-the-art medical image segmentation methods, boundary details are typically enhanced by employing complex structures, which impose an additional computational burden for inference and cannot be embedded in the latest segmentation architectures as general-purpose boundary enhancers. This paper proposes a simple and flexible method, namely, pixel-wise triplet learning, for effectively improving boundary discrimination without imposing an additional computational burden. The method uses pixel-level triplet loss to enable segmentation models to learn more discriminative feature representations at boundaries, and it can be easily incorporated into the latest segmentation networks as a generic boundary booster and used for binary and multiclass medical segmentation tasks. Extensive experiments on nine medical datasets that cover five mainstream medical imaging modalities showed that the method is simple yet effective in achieving accurate boundary discrimination and high segmentation performance. • A pixel-wise triplet learning approach to enhance boundary discrimination. • An adaptive triplets selection algorithm for fast convergence. • Validity and efficiency of the method is shown on nine medical data sets.

Semi-supervised cross-modal retrieval is an eclectic paradigm which learns common representations via exploiting underlying semantic information from both labeled and unlabeled data. Most existing methods ignore the rich semantic information of text data and are unable to fully utilize the text data in common representation learning. Moreover, they only considered the correlation of the data with the same semantic label, but ignored the correlation between the data with different semantic label. In this paper, we propose a novel semi-supervised cross-modal retrieval method, called Graph-based Semantic Alignment Network (GSAN), which learns common representation by aligning the features of different modalities with semantic embeddings of text data. Firstly, we design a Deep Supervised Semantic Encoding (DSSE) module to train the semantic projector and label predictor which can exploit the semantic embeddings and the predicted labels from unlabeled data of text modality. Then, GAN-based Bidirectional Fusion (GBF) module is designed to learn the mapping networks of two modalities (image and text). In order to make the mapping networks generate semantically discriminative and modality-invariant representations, we utilize the underlying semantic information exploited by DSSE to construct Graph-based Triplet Constraint (GTC) which can enforce feature embeddings from the semantically-matched (image and text) pairs to be more similar and push those mismatched ones away. By the benefit of fully using of semantic information, our approach can only use fewer label data and achieves the performance of state-of-the-art methods. In addition, since we only utilize the mapping networks trained in GBF module to generate common representations in referring stage, our approach is efficient and time saving in real world application. Extensive experiments on four widely-used datasets show the effectiveness of GSAN.