Cross-modality object detection is able to provide more comprehensive information about the characteristics of the target by integrating data from various modalities. However, how to simultaneously consider cross-modal interaction and global and local information modeling is still a challenging research problem. To address this issue, this paper proposes Dual-Domain Spatial-Frequency Detector(DSF-DET), a dual-domain cross-modal detector for RGB-infrared object detection. We propose the Cross-modal Large-Small (CLS) block to selectively extract image features and enhance the backbone’s multimodal feature extraction capability under complex conditions. Subsequently, we propose the Dual-Domain Feature Fusion (DDFF) block to jointly model multi-modal features from both spatial and frequency domains. Specifically, for the spatial domain, we introduce a multi-scale (MS) block to enhance spatial information at different scales. For the frequency domain, we propose the low-frequency fusion Mamba (LFM) block and the high-frequency enhancement (HFE) block to achieve global low-frequency modeling and high-frequency feature enhancement, respectively. Additionally, a Cross-scale Fusion Pyramid (CSFP) is constructed for hierarchical fusion. Extensive experiments on three public datasets demonstrate that DSF-DET has superior performance, with mAP _50 of 90.4 ^3 FD, 82.8
With the rapid proliferation of multimodal content on social media, Multimodal Aspect-Based Sentiment Analysis (MABSA) has emerged as a crucial task for fine-grained sentiment understanding. However, existing approaches face two significant challenges: the inherent representational heterogeneity between textual and visual modalities, and the prevalent interference from aspect-irrelevant visual noise in background regions. To address these limitations, we propose a unified multimodal generative framework tailored for MABSA. Specifically, we introduce a Multi-scale Modular Split Fusion (M2SF) module that refines heterogeneous features through modular decomposition, a shared attention mechanism, and dynamic block-level regularization, enabling precise semantic alignment across modalities. Additionally, we design a Globally Enhanced Multimodal Attention (GEMA) mechanism to explicitly estimate cross-modal semantic alignment and adaptively adjust attention sharpness, thereby effectively suppressing irrelevant visual noise. Extensive experiments on benchmark datasets show that our method achieves the best performance among the compared methods on MABSA and MATE while remaining competitive on MASC, validating the effectiveness and generality of the proposed framework.
Text-based person search (TBPS) is a cross-modal retrieval task matching natural language descriptions to person images. While vision-language pre-trained models have improved performance on English benchmarks, two critical challenges remain underexplored: correspondence noise in image-text pairs and multilingual applicability to low-resource languages. Correspondence noise, which manifests as either description-level mismatches or fine-grained attribute-level inconsistencies, corrupts cross-modal alignment and degrades the retrieval performance of TBPS models. Existing noise-robust methods primarily focus on description-level noise and are evaluated exclusively on English datasets, leaving attribute-level noise and non-English settings understudied. In this paper, we propose a noise-robust TBPS framework named MuCo-TBPS (Multilingual Correspondence noise correction TBPS), which integrates a multilingual vision-language pre-trained encoder (mSigLIP) with a reliability-based correspondence noise correction mechanism adopted from RDE to mitigate the impact of noisy image-text pairs. To facilitate systematic evaluation, we introduce a Noise Generator that injects both description-level and color-attribute-level noise into textual descriptions at controllable ratios, with the latter representing a fine-grained form of attribute-level correspondence noise. Extensive experiments on four benchmarks spanning English, Vietnamese, and Chinese demonstrate competitive retrieval performance across multilingual TBPS settings, while controlled-noise experiments on the English and Vietnamese benchmarks demonstrate the robustness of the proposed framework under varying correspondence-noise ratios. On the Vietnamese dataset, MuCo-TBPS outperforms TBPS-mSigLIP by 3.48 percentage points in Rank-1 accuracy and maintains strong robustness under varying noise ratios. On the English datasets, the proposed model surpasses TBPS-mSigLIP by 4.16 and 19.05 percentage points in Rank-1 on CUHK-PEDES and 3000Vn-V2E, respectively. On the Chinese PRW-TPS-CN dataset, MuCo-TBPS further improves Rank-1 by 21.84 percentage points over TBPS-mSigLIP. To the best of our knowledge, this work presents the first comprehensive study of TBPS under both description-level and color-attribute-level correspondence noise in a multilingual setting. The source code will be made available upon acceptance of the paper.
Time series class-incremental learning (TSCIL) aims to acquire new time-series classes while retaining knowledge of previously observed classes. Existing replay-free methods mainly operate on 1-D signals and therefore under-exploit complementary global morphology, including trends, periodicity, and peaks. We propose a Dual-Branch Time-series Network (DBTNet) that jointly encodes raw 1-D sequences and deterministically rendered 2-D graph images. Instead of the isolated reuse of image rendering, knowledge distillation, or parameter importance, the key contribution is their TSCIL-specific coupling: the temporal branch anchors local dynamics, the visual branch supplies complementary global structure, cross-modal gating controls their interaction, and asymmetric hybrid distillation together with selective parameter regularization constrains representation drift, decision-boundary drift, and parameter overwriting without replaying old raw samples. Experiments on four sensing benchmarks consistently demonstrate higher mean average accuracy and lower mean forgetting than previous replay-free methods evaluated under the same protocols. Ablation experiments further validate the effectiveness of the components used in DBTNet.
Dual-encoder vision-language models (VLMs) allow to embed texts and images in a shared representation space. However, it has been shown that these models are subject to a “modality gap phenomenon” meaning there exists a clear separation between the embeddings from one modality and another in the embedding space. While this misalignment is detrimental for downstream tasks such as multimodal retrieval, multimodal clustering or zero-shot classification, no generic and practical methods have so far been proposed to assess it precisely and even reduce it. We therefore propose novel measures and an effective spectral technique to achieve this goal, noting that, in its basic form, the spectral method offers no out-of-sample extension – although a Nyström-style approximation could provide one – and mainly targets offline retrieval settings. On the other hand, a new breed of model has emerged, that can handle combined-modal inputs to a multimodal large language model, and extract a unified, fused vector representation. These models open up new research questions about possible biases of these fused representations towards a specific modality, and should such biases be found to exist, if they can be reduced drawing on the same techniques utilized for dual-encoder models.
