Medical image segmentation is challenging due to limited annotated data, high labeling costs, and substantial image heterogeneity. Although large-scale vision foundation models (e.g., SAM) have shown great potential in this field, existing SAM-based methods typically rely on expert-defined geometric prompts or complex clinical text prompts, which limits their generalizability across diverse medical image segmentation tasks. To overcome these challenges, we propose Simple-ViLMedSAM, a CLIP-SAM integration framework that enables high-accuracy segmentation in zero-shot and few-shot settings using only simple text queries, that is, using only basic anatomical or disease-related text labels. At its core is an Implicit Pos-Prompter (IPP), which generates attribution maps containing implicit positional cues to replace traditional geometric prompts. IPP incorporates a multi-modal information bottleneck and an affinity-based refinement strategy to ensure high-quality guidance from CLIP-SAM interactions. To further enhance segmentation, we introduce a Bidirectional Interaction Decoder (BID) that employs bidirectional cross-attention to align IPP’s positional maps with SAM's pixel-level features. By jointly modeling global semantics and local details, BID significantly improves segmentation accuracy. Extensive experiments on four public datasets demonstrate that Simple-ViLMedSAM consistently outperforms existing methods in both zero-shot and few-shot medical image segmentation tasks, using only simple text queries. The code will be publicly available upon acceptance.

The human cerebral cortex encodes rich neurobiological information that is essential for understanding brain development, aging, and disease. Although various cortical representation learning methods have been proposed, existing models are typically restricted to stage-specific cohorts and lack generalization across the lifespan. While recent vision-language models offer a promising direction, building a unified framework for cortical representation faces three key challenges: (1) the non-Euclidean manifold structure of cortical surfaces, (2) homogenization of individual folding patterns induced by registration, and (3) distribution shifts of cortical features across the lifespan. To address these issues, we present CortiLife, the first unified vision-language framework for lifespan-aware cortical representation learning. Specifically, CortiLife introduces a surface tokenizer that integrates icosahedron-based surface patchification with multi-level patch encoding to transform complex cortical manifolds into compact token representations. The multi-level encoding incorporates three complementary streams that capture local topology, global interactions, and patch-wise distributional patterns, effectively mitigating the challenges of homogenization and distribution shifts. Furthermore, CortiLife integrates masked self-distillation with metadata language prompting, embedding information such as age, sex, health status, and attribution type into the text encoder to better capture individual-specific cortical representations while enabling both age-aware and modality-aware modeling. Extensive experiments on downstream tasks, including two encoder-frozen tasks (age prediction and cortical parcellation) and four encoder fine-tuning tasks (brain disorder diagnosis), demonstrate that CortiLife consistently outperforms state-of-the-art baselines across different age stages and modality types, underscoring its effectiveness and generalization ability.

Glaucoma is a leading cause of irreversible blindness globally. When glaucoma is diagnosed, Anterior Chamber Angle (ACA) evaluation is the necessary step for the prognosis and treatment of glaucoma. However, current clinical evaluation methods are labor intensive and rely on expert judgment, which makes them inefficient. Automating ACA classification based on images using machine learning, especially deep neural networks, holds promise. Yet, image samples alone can't provide sufficient high-level semantic information on ACA, resulting in suboptimal classification performance. This paper proposes a novel end-to-end knowledge-guided bi-modal network (KGNet) for ACA evaluation. Specifically, we consider two modalities of ACA data: textual domain knowledge and images. We first design a new strategy to refine class-based knowledge into textual descriptions, thereby increasing the diversity of features learned by the model. We then extract two types of representations using two distinct components: 1) a supervised loss is applied to learn modality-specific representations by incorporating domain knowledge; 2) a fusion module that uses knowledge-guided learning to highlight key clinical structures in ACA images leveraging bimodal correlations. Experimental results on an ACA dataset and three public datasets show that our method outperforms several state-of-the-art deep learning models in eye image evaluation, indicating the potential medical interest of our method. Furthermore, our approach improves interpretability by explicitly aligning visual representations with structured clinical knowledge, enabling more structured and clinically grounded explanations than conventional models.

Electroencephalography (EEG) has become an important tool for cognitive state assessment. Although deep learning methods have achieved remarkable progress in this area, most existing approaches overlook the heterogeneous encoding patterns across different brain regions and the multi-scale dynamics of neural activity, which limits the reliability and generalization of EEG representations. To address this issue, we propose MSTA-EEGNet, a multi-scale task-aware network for EEG-based cognitive state recognition. In this framework, EEG channels are partitioned into multiple functional regions, with independent encoders assigned to each region to capture distinct neural patterns. A multi-scale feature extraction strategy is further introduced to model neural dynamics across different time resolutions. Furthermore, a task-aware regional feature fusion module is designed to adaptively emphasize task-relevant regions and guide cross-region feature integration, thereby enhancing task-specific representations. Extensive experiments on emotion recognition, workload assessment, and motor imagery demonstrate that MSTA-EEGNet consistently outperforms representative state-of-the-art methods. The codes are available in https://github.com/Codsworth-2001/MSTA-EEGNet.

Automated radiology report generation (RRG) aims to produce detailed textual reports from clinical imaging, such as computed tomography (CT) scans, to improve the accuracy and efficiency of diagnosis and provision of management advice. RRG is complicated by two key challenges: (1) inherent complexity in extracting relevant information from imaging data under resource constraints, and (2) difficulty in objectively evaluating discrepancies between model-generated and expert-written reports. To address these challenges, we propose μ ^2 LLM, a mu ltiscale mu ltimodal large language models for RRG tasks. The novel μ^2 Tokenizer, as an intermediate layer, integrates multi-modal features from the multiscale visual tokenizer and the text tokenizer, then enhances report generation quality through direct preference optimization (DPO), guided by GREEN-RedLlama. Experimental results on four large CT image-report medical datasets demonstrate that our method outperforms existing approaches, highlighting the potential of our fine-tuned μ ^2 LLMs on limited data for RRG tasks. All code, data, and models will be publicly available in our official repository: https://github.com/Siyou-Li/u2Tokenizer .

Generalized Zero-Shot Learning (GZSL) addresses the challenge of classifying unseen classes in the presence of seen classes by leveraging semantic attributes to bridge the gap for unseen classes. However, in image based disease classification, such as glioma sub-typing, distinguishing between classes using image semantic attributes can be challenging. To address this challenge, we introduce a novel GZSL method that eliminates the dependency on semantic information. Specifically, we propose that the primary of most classification in clinic is risk stratification, and classes are inherently ordered rather than purely categorical. Based on this insight, we present an inter-class feature augmentation (IFA) module, where distributions of different classes are ordered by their risk levels in a learned feature space using pre-defined joint conditional Gaussian distribution model. This ordering enables the generation of unseen class features through feature mixing of adjacent seen classes, effectively transforming the zero-shot learning problem into a supervised learning task. Our method eliminates the need for explicit semantic information, avoiding the cross-modal alignment between visual and semantic features. Moreover, the IFA module for GZSL requires no structural modifications to the existing classification models. In the experiment, both in-house and public datasets are used to evaluate our method across different tasks, including glioma subtyping, Alzheimer’s disease (AD) classification and diabetic retinopathy classification. Experimental results demonstrate that our method outperforms the state-of-the-art GZSL methods with statistical significance.

Retinopathy of prematurity (ROP) is a potentially blinding retinal disease that often affects low birth weight premature infants. Lesion detection and recognition are crucial for ROP diagnosis and clinical treatment. However, this task poses challenges for both ophthalmologists and computer-based systems due to the small size and subtle nature of many ROP lesions. To address these challenges, we present a Sequence encoding and Block balancing-based Segmentation Network (SeBSNet), which incorporates domain knowledge coding, sequence coding learning (SCL), and block-weighted balancing (BWB) techniques into the segmentation of ROP lesions. The experimental results demonstrate that SeBSNet outperforms existing state-of-the-art methods in the segmentation of ROP lesions, with average ROC_AUC, PR_AUC, and Dice scores of 98.84%, 71.90%, and 66.88%, respectively. Furthermore, the integration of the proposed techniques into ROP classification networks as an enhancing module leads to considerable improvements in classification performance.

Semi-supervised learning (SSL) can effectively reduce the labor-intensive labeling required for deep learning based medical image segmentation. The emergence of visual foundation models show zero-shot capability, offering a new way of SSL. In this paper, a novel SSL framework that combines foundation and dedicated models is proposed. Unlike most existing SSL methods, where the foundation model is manually prompted to generate pseudo-labels from unlabeled images for training the dedicated model in a one-way strategy without further refinement. In our framework, foundation (SAM2) and dedicated (UNet) models are in an iterative pipeline. Specifically, in each iteration, prompts from coarse segmentation results using UNet are calculated for SAM2 to generate pseudo-labels which are used to further train the UNet for better prompts in next iteration. In this way, the pseudo-labels and UNet can be mutually improved until convergence. To enhance the performance of SAM2 in medical image segmentation, a new uncertainty-aware module using historical cues is presented to optimize key frames selection and prompts generation for SAM2. Furthermore, a new semantic-aware memory bank is introduced, where memories in the memory bank of SAM2 are divided into semantic groups. In this way, anatomical prior knowledge can be leveraged by SAM2. In the experiment, our framework is evaluated using public and in-house datasets in the context of multi-label segmentation, and the experimental results demonstrate that our framework outperforms state-of-the-art SSL methods in both datasets.

Decoding visual stimuli from human brain activity is a fundamental challenge in cognitive neuroscience and neuroimaging. While recent advances in deep learning have significantly improved the performance of fMRI-to-image decoding, most existing methods overlook the issue of inter-subject variability in fMRI data, which leads to poor generalization across subjects. Current approaches often rely on partially shared model architectures that offer limited generalization and still require subject-specific components, restricting their applicability to unseen subjects. To address this limitation, we propose BrainX, a universal brain decoding framework that constructs a unified fMRI encoder and image generator to achieve subject-agnostic modeling. Specifically, we introduce a feature disentanglement mechanism that extracts subject-shared features from the fMRI embeddings, which are then fed into the image generator to reconstruct visual stimuli. This design eliminates the need for subject-specific models and significantly enhances cross-subject generalization. Additionally, we develop a neuro-geometric fMRI representation learning method that projects 3D cortical structures onto a 2D surface space, effectively mitigating the inaccuracies caused by imprecise geodesic distance estimation in 3D Euclidean space. Extensive experiments on the Natural Scenes Dataset (NSD) demonstrate that BrainX consistently outperforms existing state-of-the-art methods across three decoding settings: within-subject, cross-subject with finetuning, and cross-subject without finetuning. The codes is available at https://github.com/WENXUYUN/BrainX.

BACKGROUND:The main task of deep learning (DL) based brain tumor segmentation is to get accurate projection from learned image features to their corresponding semantic labels (i.e., brain tumor sub-regions). To achieve this goal, segmentation networks are required to learn image features with high intra-class consistency. However, brain tumor are known to be heterogeneous, and it often causes high diversity in image gray values which further influences the learned image features. Therefore, projecting such diverse image features (i.e., low intra-class consistency) to the same semantic label is often difficult and inefficient. PURPOSE:The purpose of this study is to address the issue of low intra-class consistency of image features learned from heterogeneous brain tumor regions and ease the projection of image features to their corresponding semantic labels. In this way, accurate segmentation of brain tumor can be achieved. METHODS:We propose a new DL-based method for brain tumor segmentation, where a semantic feature module (SFM) is introduced to consolidate image features with meaningful semantic information and enhance their intra-class consistency. Specifically, in the SFM, deep semantic vectors are derived and used as prototypes to re-encode image features learned in the segmentation network. Since the relatively consistent deep semantic vectors, diversity of the resulting image features can be reduced; moreover, semantic information in the resulting image features can also be enriched, both facilitating accurate projection to the final semantic labels. RESULTS:In the experiment, a public brain tumor dataset, BraTS2022 containing, multi-sequence MR images of 1251 patients is used to evaluate our method in the task of brain tumor sub-region segmentation, and the experimental results demonstrate that, benefiting from the SFM, our method outperforms the state-of-the-art methods with statistical significance ( p < 0.05 $p<0.05$ using the Wilcoxon signed rank test). Further ablation study shows that the proposed SFM can yield an improvement in segmentation accuracy (Dice index) of up to 11% comparing with that without the SFM. CONCLUSIONS:In DL-based segmentation, low intra-class consistency of learned image features degrades segmentation performance. The proposed SFM can effectively enhance the intra-class consistency with high-level semantic information, making the projection of image features to their corresponding semantic labels more accurate.

Decoding natural language text from non-invasive brain signals, such as functional magnetic resonance imaging (fMRI), remains a central challenge in brain-computer interface research. While recent advances in large language models (LLMs) have enabled open-vocabulary fMRI-to-text decoding, existing frameworks typically process the entire fMRI sequence in a single step, leading to performance degradation when handling long input sequences due to memory overload and semantic drift. To address this limitation, we propose a brain-inspired sequential fMRI-to-text decoding framework that mimics the human cognitive strategy of segmented and inductive language processing. Specifically, we divide long fMRI time series into consecutive segments aligned with optimal language comprehension length. Each segment is decoded incrementally, followed by a wrap-up mechanism that summarizes the semantic content and incorporates it as prior knowledge into subsequent decoding steps. This sequence-wise approach alleviates memory burden and ensures semantic continuity across segments. In addition, we introduce a text-guided masking strategy integrated with a masked autoencoder (MAE) framework for fMRI representation learning. This method leverages attention distributions over key semantic tokens to selectively mask the corresponding fMRI time points, and employs MAE to guide the model toward focusing on neural activity at semantically salient moments, thereby enhancing the capability of fMRI embeddings to represent textual information. Experimental results on the two datasets demonstrate that our method significantly outperforms state-of-the-art approaches, with performance gains increasing as decoding length grows.

Scene text editing aims to modify or add texts on images while ensuring text fidelity and overall visual quality consistent with the background. Recent methods are primarily built on UNet-based diffusion models, which have improved scene text editing results, but still struggle with complex glyph structures, especially for non-Latin ones (, Chinese, Korean, Japanese). To address these issues, we present FLUX-Text, a simple and advanced multilingual scene text editing DiT method. Specifically, our FLUX-Text enhances glyph understanding and generation through lightweight Visual and Text Embedding Modules, while preserving the original generative capability of FLUX. We further propose a Regional Text Perceptual Loss tailored for text regions, along with a matching two-stage training strategy to better balance text editing and overall image quality. Benefiting from the DiT-based architecture and lightweight feature injection modules, FLUX-Text can be trained with only 0.1M training examples, a 97% reduction compared to 2.9M required by popular methods. Extensive experiments on multiple public datasets, including English and Chinese benchmarks, demonstrate that our method surpasses other methods in visual quality and text fidelity. All the code is available at https://github.com/AMAP-ML/FluxText.

By pretraining on trillions of tokens, an LLM gains the capability of text generation. However, to enhance its utility and reduce potential harm, SFT and alignment are applied sequentially to the pretrained model. Because SFT and alignment have different objectives and underlying processes, performance on certain tasks can decline. To address this, we seamlessly introduce Unified Fine-Tuning (UFT), which integrates SFT and alignment into a single training stage using the same objective and loss functions through an implicit reward function. Our experimental results demonstrate that UFT outperforms SFT on instruction-tuning data alone. Moreover, when combining instruction-tuning data with alignment data, UFT effectively prevents the degradation on some tasks across these two stages and shows a clear advantage over sequentially applying SFT and alignment. This is evident in the significant improvements observed in the ifeval task for instruction-following and the truthful task for factuality. The proposed general fine-tuning framework UFT establishes an effective and efficient paradigm for LLM post-training.

In recent years, deep learning has been successfully adopted in a wide range of applications related to electronic health records (EHRs) such as representation learning and clinical event prediction. However, due to privacy constraints, limited access to EHR becomes a bottleneck for deep learning research. To mitigate these concerns, generative adversarial networks (GANs) have been successfully used for generating EHR data. However, there are still challenges in high-quality EHR generation, including generating time-series EHR data and imbalanced uncommon diseases. In this work, we propose a Multi-label Time-series GAN (MTGAN) to generate EHR and simultaneously improve the quality of uncommon disease generation. The generator of MTGAN uses a gated recurrent unit (GRU) with a smooth conditional matrix to generate sequences and uncommon diseases. The critic gives scores using Wasserstein distance to recognize real samples from synthetic samples by considering both data and temporal features. We also propose a training strategy to calculate temporal features for real data and stabilize GAN training. Furthermore, we design multiple statistical metrics and prediction tasks to evaluate the generated data. Experimental results demonstrate the quality of the synthetic data and the effectiveness of MTGAN in generating realistic sequential EHR data, especially for uncommon diseases.

Deep learning based multi-atlas segmentation (DL-MA) has achieved the state-of-the-art performance in many medical image segmentation tasks, e.g., brain parcellation. In DL-MA methods, atlas-target correspondence is the key for accurate segmentation. In most existing DL-MA methods, such correspondence is usually established using traditional or deep learning based registration methods at image level with no further feature level adaption. This could cause possible atlas-target feature inconsistency. As a result, the information from atlases often has limited positive and even counteractive impact on the final segmentation results. To tackle this issue, in this paper, we propose a new DL-MA framework, where a novel differentiable atlas feature warping module with a new smooth regularization term is presented to establish feature level atlas-target correspondence. Comparing with the existing DL-MA methods, in our framework, atlas features containing anatomical prior knowledge are more relevant to the target image feature, leading the final segmentation results to a high accuracy level. We evaluate our framework in the context of brain parcellation using two public MR brain image datasets: LPBA40 and NIREP-NA0. The experimental results demonstrate that our framework outperforms both traditional multi-atlas segmentation (MAS) and state-of-the-art DL-MA methods with statistical significance. Further ablation studies confirm the effectiveness of the proposed differentiable atlas feature warping module.

Automatic medical report generation (MRG), which aims to produce detailed text reports from medical images, has emerged as a critical task in this domain. MRG systems can enhance radiological workflows by reducing the time and effort required for report writing, thereby improving diagnostic efficiency. In this work, we present a novel approach for automatic MRG utilizing a multimodal large language model. Specifically, we employed the 3D Vision Transformer (ViT3D) image encoder introduced from M3D-CLIP to process 3D scans and use the Asclepius-Llama3-8B as the language model to generate the text reports by auto-regressive decoding. The experiment shows our model achieved an average Green score of 0.3 on the MRG task validation set and an average accuracy of 0.61 on the visual question answering (VQA) task validation set, outperforming the baseline model. Our approach demonstrates the effectiveness of the ViT3D alignment of LLaMA3 for automatic MRG and VQA tasks by tuning the model on a small dataset.

Structural magnetic resonance imaging characterizes the morphology and anatomical features of the brain and has been widely utilized in the diagnosis of developmental disorders. Given the dynamic nature of developmental disorder progression with age, existing methods for disease detection have incorporated age as either prior knowledge to be integrated or as a confounding factor to be disentangled through supervised learning. However, the excessive focus on age information in these methods restricts their capability to unearth disease-related features, thereby affecting the subsequent disease detection performance. To address this issue, this work introduces a novel weakly supervised learning-based method, namely, the Weakly Supervised Spherical Age Disentanglement Network (WSSADN). WSSADN innovatively combines an attention-based disentangler with the Conditional Generative Adversarial Network (CGAN) to remove normal developmental information from the brain representation of the patient with developmental disorder in a weakly supervised manner. By reducing the focus on age information during the disentanglement process, the effectiveness of the extracted disease-related features is enhanced, thereby increasing the accuracy of downstream disease identification. Moreover, to ensure effective convergence of the disentanglement and age information learning modules, we design a consistency regularization loss to align the age-related features generated by the disentangler and CGAN. We evaluated our method on three different tasks, including the detection of preterm neonates, infants with congenital heart disease, and autism spectrum disorders. The experimental results demonstrate that our method significantly outperforms existing state-of-the-art methods across all tasks. The codes will be publicly available in https://github.com/xuepengcheng1231/WSSADN.

The mask autoencoder (MAE) is utilized in functional magnetic resonance imaging (fMRI) analysis to construct brain representation learning models and conduct prediction for various fMRI-related tasks (e.g., disease detection). It involves pretraining the model by reconstructing signals of brain regions that are randomly masked at different time segments and subsequently fine-tuning it for prediction tasks. Although the MAE helps to improve prediction performance, directly applying it to fMRI may lead to sub-optimal results for the following reasons: 1) The reconstruction process is not task-aware, meaning the extracted brain representations are unable to sufficiently consider downstream tasks, thereby affecting prediction performance; 2) Random masking of fMRI data ignores that the varying contributions of different brain regions to different prediction tasks. To address these issues, we propose Task-Aware Reconstruction Dynamic Representation Learning (TARDRL). Different from the conventional sequential design, this approach sets up reconstruction and prediction tasks in parallel to learn robust task-aware representations. Based on the parallelized framework, we leverage attention maps from specific tasks to guide the fMRI time series reconstruction, which in turn helps to learn task-aware fMRI representations and improve disease prediction accuracy. Extensive experiments demonstrate that our model outperforms state-of-the-art methods on the ABIDE and ADNI datasets, with high interpretability. The codes are available in the repository.

Medical imaging plays a critical role in various clinical applications. However, due to considerations such as cost and risk, the acquisition of certain image modalities can be limited. To address this issue, many cross-modality medical image synthesis methods have been proposed. Nevertheless, current methods struggle to accurately model hard-to-synthesize regions (e.g., tumor or lesion regions). To overcome this challenge, we propose a simple yet effective strategy: a dual-discriminator (dual-D) adversarial learning system. In this system, 1) a global discriminator (global-D) provides an overall evaluation of the synthetic image, and 2) a local discriminator (local-D) performs a dense evaluation of the synthetic image’s local regions. Additionally, we introduce a difficult-region-aware attention mechanism that enhances the modeling of hard-to-synthesize regions (e.g., tumor or lesion regions) based on the local-D. Experimental results demonstrate the robustness and accuracy of our proposed method in synthesizing target images from corresponding source images. Specifically, we evaluated our method on two datasets: i.e., 1) generating T2 MRI from T1 MRI for brain tumor images, and 2) generating CT from MRI. Our proposed method outperforms state-of-the-art techniques in both datasets and tasks. Furthermore, our proposed difficult-region-aware attention mechanism proves effective in generating more realistic images, particularly in the hard-to-synthesize regions.