Multi-modality data has been widely used in remote sensing image semantic segmentation (RSISS) owing to the complementarity between modalities. However, different modalities are hardly obtained at the same time in most practical application scenarios. In the presence of modality missing, traditional multimodal learning methods suffer from accuracy degradation, caused by the feature distribution shift resulting from the lack of complementary information. To address this problem, we propose a multi-to-uni modality knowledge distillation method to transfer knowledge contained in the multimodal teacher to unimodal student, which leverages existing multimodal data to improve the accuracy under unimodal conditions. This method includes the modality-enhanced fusion (MEF) module, the frequency-awareness feature distillation (FAD) module and the cross-modality affinity distillation (CAD) module. The MEF aims at integrating multimodal features and reducing modality discrepancy between the teacher and student network by enhancing the mutual modality during fusion. The FAD module leverages the discrete wavelet transformation (DWT) to obtain high-frequency maps from low-level features, which are subsequently transferred to the unimodal student as detailed knowledge. The CAD module transfers high-level structural knowledge to student by constructing affinity maps between modalities. Extensive experiments on two commonly used datasets demonstrate that the proposed multi-to-uni modality distillation method outperforms the state-of-the-art methods.

Semisupervised deep facial expression recognition (FER) tries to learn better representations from both labeled and unlabeled data to avoid the huge manual labeling cost in the supervised methods. Due to the imbalanced distribution of facial expression data and the varying difficulty in recognizing different expressions, conventional semisupervised FER methods that directly discard low-confidence unlabeled samples may exacerbate the deficiency of the minority classes, and trap the model into the Matthew effect. Therefore, an adaptive contrastive learning-based semisupervised deep facial expression recognition framework is proposed to utilize both high and low confidence unlabeled samples by two different contrastive learning strategies according to an adaptive threshold. The threshold varies from different classes and iterations, considering both the difficulty of different classes and current learning state of the model during the training process. After the adaptive thresholding, the class-aware contrastive learning is applied to the high confidence samples, while the distance-aware contrastive learning is to the low confidence ones. Our proposed method achieves the state-of-the-art performance through extensive experiments on two widely-used datasets RAF-DB and AffectNet.

Referring remote sensing image segmentation (RRSIS) aims to map natural language queries to pixel-level masks within complex geographic scenes, requiring precise cross-modal semantic alignment. In recent years, Mamba has demonstrated impressive performance in various remote sensing tasks due to its powerful contextual modeling capability, offering new potential for RRSIS. However, vanilla Mamba primarily models sequential dependencies and provides limited support for cross-modal interaction, thus hindering precise vision–language semantic alignment. To overcome this limitation, we propose M3RIS, a mutual modulation Mamba network for RRSIS. Specifically, we propose the vision–language mutual modulation (VLMM) module that distills semantic knowledge from the Segment Anything Model (SAM) to provide auxiliary semantic cues for cross-modal interaction. The module not only uses the distilled knowledge to guide the prediction of channel-wise affine coefficients and spatial dynamic convolution kernels to enhance target-relevant visual responses, but also introduces this knowledge into cross-attention as a semantic bias to enable language features to aggregate semantically relevant visual context. This facilitates mutual enhancement between vision and language, promoting more effective cross-modal alignment than conventional unidirectional modulation. Moreover, we introduce the hybrid semantic refined (HSR) decoder, which alternates between visual state space (VSS) blocks and cross-modal semantic refinement (CSR) modules. The VSS blocks capture global semantics via long-range context modeling, while the CSR modules derive multi-scale dynamic convolution kernels from language features to refine local visual features through dynamic filtering, thereby enhancing semantic awareness of the referred object. Extensive experiments on RRSIS-D, RISBench, and RefSegRS datasets demonstrate that M3RIS outperforms several previous state-of-the-art approaches.

Source-free object detection aims to adapt a source-trained model to an unlabeled target domain without source data. Existing methods typically employ pseudo labeling paradigm with a teacher-student framework, where the quality of pseudo labels heavily depends on the confidence threshold. Selecting an optimal threshold is challenging due to confidence distributions differing across categories and category imbalance. Therefore, a dual-adapter pseudo labeling method is proposed to obtain more reliable pseudo labels for source-free object detection. Specifically, the proposed method works with two adapters, i.e., Category Confidence Adapter (CCA) and Category Imbalance Adapter (CIA). CCA is designed to dynamically adjust the confidence range of each category according to a statistically guided boundary calibration mechanism, aiming to obtain candidate pseudo labels. The median of these candidate pseudo labels is used as the current confidence threshold for each category. CIA updates the confidence threshold of each category by fusing its historical and current estimates, whose weights are dynamically computed based on the category frequency. By adaptively estimating and updating thresholds across categories, the two adapters alleviate category-specific confidence distribution differences and category imbalance in the target domain, thereby improving the reliability of pseudo labels. The proposed method demonstrates competitive performance across multiple benchmark datasets, indicating its effectiveness in cross-domain object detection.

Domain generalization, semantic segmentation of optical remote sensing images, aims to train a model on one or multiple source domains such that it can generalize to unseen target domains without accessing target-domain data during training. Existing methods are mainly divided into style randomization (SR) and feature normalization (FN). These methods encourage the model to focus more on domain-invariant information by reducing the impact of style information on the model training. However, owing to the entanglement between style information and domain-invariant information, SR tends to partially change domain-invariant information, while FN inevitably leads to the partial loss of domain-invariant information. To address these limitations, we propose style-shift-aware category contrastive learning (SSAC), which can further explore the domain-invariant information embedded in the source domain. Then, the domain-invariant information is leveraged to compensate for the regions impacted by SR. Furthermore, to enrich the style diversity of the auxiliary domain, we introduce a dual-branch augmentation (DBA) strategy. Strong augmentations first simulate variations caused by different image times, followed by a dual-branch design that simulates resolution-related differences caused by different image equipment: RandConv simulates resolution degradation, while an edge-texture augmentation (ETA) module simulates resolution enhancement. Extensive experiments show that our method consistently outperforms state-of-the-art (SOTA) approaches. It achieves at least 6.22/5.03 mIoU gains on the WHU dataset and 3.48/5.78 mIoU gains on the OM dataset under ResNet-50/101, attaining the highest average performance across subsets. Our code could be available at https://github.com/yuan3ee/SSAC

Unsupervised domain-adaptive object detection (UDA-OD) in remote sensing images aims to transfer knowledge from a labeled source domain to an unlabeled target domain, mitigating performance degradation caused by domain shift. Existing methods commonly adopt a teacher-student framework, where a single teacher, updated via an exponential moving average (EMA) of the student parameters, generates pseudolabels for training. While this framework effectively transfers knowledge from the teacher to the student, the quality of pseudolabels and the alignment of source- and target-domain features remain key challenges for reducing domain shift. To address these challenges, a dual-teacher prototype consistency (DTPC) method is proposed. Specifically, DTPC employs two available object detection models as teachers to provide complementary guidance for the student. The larger teacher, equipped with a deeper backbone and kept frozen, extracts domain-invariant prototype anchors through a class-aware prototype update (CAPU) mechanism, thereby facilitating effective feature-level alignment between the source and target domains. Furthermore, the larger teacher and an EMA-updated smaller teacher jointly generate pseudolabels to supervise the student with soft- and hard-label losses, improving the quality and reliability of pseudolabels. Extensive experiments on two remote sensing benchmarks demonstrate that DTPC achieves the state-of-the-art (SOTA) results, surpassing the second-best methods by 4.6% and 4.3% mean average precision (mAP) on the NWPU VHR-10 -> ightarrow detectIon in optical remote sensing images (DIOR) and VisDrone -> DroneVehicle adaptation settings, respectively.

Unsupervised domain adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain to improve model generalization, and has been widely used in remote sensing image (RSI) semantic segmentation. However, complex intradomain category distributions and mismatched cross-domain category priors hinder accurate domain alignment in existing UDA methods. To address these challenges, we propose a probabilistic multiple prototypes contrastive alignment (PMPCA) UDA method. First, we develop a probabilistic multiple prototypes learning (PMPL) module, which constructs multiple prototypes along with probabilistic parameters to represent both the central tendency and dispersion of category distributions. This module enables fine-grained preservation of category semantics in both the source and target domains. Second, we introduce a prior-weighted contrastive (PWC) constraint, which establishes prototype-level contrastive learning between the source and target domains. It pulls prototypes toward their corresponding categories while pushing them away from other categories, thereby promoting structural alignment across domains. Furthermore, we estimate category frequencies in both domains and use them as prior weights for adaptive calibration, mitigating bias induced by category prior and refining alignment. Finally, we design an uncertainty-guided curriculum adaptation (UCA) strategy, which quantifies the difficulty of the target domain based on prediction uncertainty and enables progressive UDA from easy to hard cases. This strategy reduces the interference of noisy pseudo-labels. Extensive experiments on six UDA settings demonstrate that PMPCA consistently outperforms state-of-the-art methods and generalizes well across spatial regions and spectral bands.

Remote sensing image captioning (RSIC) aims to describe high-level semantics, spatial configurations, and relationships between objects in remote sensing images through natural language. Recent advances have attempted to incorporate entity information from remote sensing images to guide the generation of accurate captions. Although these methods have shown promising results, their effectiveness remains limited due to low entity recognition accuracy and insufficient utilization of the extracted entity information. To address these issues, we propose a dual-stream framework, namely entity-guided caption generation for remote sensing images (EG-Cap). Specifically, the EG-Cap first employs a sequential multientity generator (SMEG) to reformulate entity recognition as a sequence generation task, improving recognition precision by capturing entity contextual dependencies. It then uses an entity-vision adaptive fusion (EVAF) module, via an entity-aware gating mechanism, to adaptively fuse entities with visual features by weighting their semantic compatibility, thereby enhancing the utilization of entity information. Finally, a multiscale contextual aggregation (MCA) module leverages a cascaded Transformer-based decoder to capture multiscale contextual dependencies from the entity-visual representation, ensuring semantically coherent caption generation. Experimental results on the Sydney-Captions, UCM-Captions, and RSICD datasets demonstrate the superior performance of EG-Cap compared to the existing state-of-the-art methods. The implementation code of EG-Cap will be publicly available at: https://github.com/Xiliang-Li/EG-Cap

Remote sensing image captioning (RSIC) aims to generate natural language descriptions for complex remote sensing imagery through effective cross-modal alignment between visual content and textual semantics. Due to densely distributed objects and complex spatial layouts, remote sensing scenes exhibit strong dependencies between local object details and global semantic context, posing significant challenges for discriminative visual representation learning. Existing methods struggle to capture such global–local dependencies and conduct targeted enhancement for key local regions, leading to fragmented global–local representations and limited semantic discriminability in generated captions. To address this issue, the Anchor-Guided Spatial-Semantic Modeling (AGSM) is proposed for RSIC. AGSM uses global–local semantic dependencies from CLIP-extracted CLS and patch tokens to guide feature enhancement and interaction. Specifically, a Visual Anchor Feature Diffusion (VAFD) module strengthens representative local features through anchor-based aggregation and diffusion. A Bi-Mamba Spatial Infusion (BMSI) module injects spatial dependencies into global semantic representations via bidirectional sequence modeling. Furthermore, a Spatially Gated FiLM-Mamba (SGFM) module, where FiLM denotes Feature-wise Linear Modulation, enables adaptive and cooperative global–local interaction through spatially adaptive modulation. The fused representations are finally decoded by a Mamba-based language decoder to generate accurate and coherent captions. Experimental results on four public benchmark datasets, including Sydney-Captions, UCM-Captions, RSICD, and NWPU-Captions, demonstrate the effectiveness and competitive performance of the proposed method.

Remote sensing image super-resolution (SR) technology can provide more accurate data support for environmenremove tal monitoring, disaster assessment, and other fields. Recently, Mamba-based methods have achieved excellent results in the SR field, outperforming Convolutional Neural Networks (CNNs) and Transformers. However, these methods may suffer from texture misalignment and distortion since the scanning mechanism in Mamba may destroy the inherent two-dimensional spatial continuity of the images. Moreover, the texture details may be blurred or lost due to the limited ability of its state transition paradigm to model high-frequency details. To adremove dress the above two drawbacks, we propose a new Mamba-Enhanced Local Attention network (MELA). It utilizes Mamba and overlapping window attention mechanisms to achieve global-local feature complementarity, effecremove tively enhancing high-frequency details and adaptively modeling feature associations. Specifically, we use the proposed Local Self-Attention Vision Mamba (LSViM) in MELA to enhance spatial modeling capabilities while maintaining computational efficiency through the synergy of the cross-scanning layer and the attention mechremove anism. Experimental results on 4 widely used remote sensing datasets demonstrate that MELA outperforms the existing methods, and obtains high-resolution images with clearer edges and stronger feature continuity.

Scene Graph Generation (SGG), which transforms an image into a knowledge graph of objects and their relationships, has great potential in downstream tasks such as image captioning and visual question answering. However, existing SGG models struggle with the long-tailed distribution of predicates, where infrequent tail predicates are severely underrepresented, leading models to overfit frequent head predicates and overlook more informative tail ones. To address this challenge, we propose a novel data augmentation approach, Multimodal Large Language Model-Driven Pseudo-labeling (MLLMP), designed to enrich the supervision of tail predicates. Our method first detects object pairs with foreground relationships and generates candidate predicate labels via MLLM-based region descriptions. A dynamic candidate label selection module then adaptively regulates label assignment to alleviate data imbalance, while a self-validation strategy further refines the final pseudo-labels. Due to the model-agnostic nature of our method, MLLMP can be seamlessly integrated into various SGG models. Extensive experiments on multiple benchmarks demonstrate its effectiveness and consistent improvements, particularly on tail predicates, thus advancing the robustness and practicality of SGG.

The fusion of hyperspectral image (HSI) and multispectral image (MSI) aims to fully exploit the complementary advantages of both modalities to generate a high-resolution HSI. However, existing HSI fusion methods fail to fully utilize frequency-domain information when capturing deep correlations between spectral bands, leading to difficulties in accurately preserving spatial texture details. Furthermore, these methods often overlook the intrinsic differential characteristics in HSI fusion, which easily causes spatial-spectral ( S-2 ) decoupling distortions. To address the aforementioned challenges, this article proposes a S-2 differential feature awareness network (SSDAN) that employs a multiscale input strategy to fully preserve the rich spatial structures in the source images. Specifically, a spatial-frequency fusion network (SFFN) with a dual-branch architecture is designed to separately model multilevel spatial-frequency features of HSI and MSI, thereby effectively enhancing the joint representation of high-frequency details and global-local structures. In addition, we introduce a differential S-2 complementarity module (DSSCM) that explicitly models modality-specific detailed features and adaptively aligns the dual-branch fused features with the shallow features, effectively mitigating decoupling distortions caused by feature mismatches. The code will be available at https://github.com/MSFLabX/SSDAN

Multi-modal image matching is a fundamental task in computer vision that has made significant progress. Due to modality changes and geometric distortions, the distinctiveness of adjacent descriptors and the precision of keypoint positions may not be sufficient enough to minimize the position errors between matched feature points. Previous works refine matches by either adjusting positions using an initial transformation matrix with a hard elimination threshold or regressing local sub-pixel coordinates supervised by the symmetric epipolar distance function. However, their performance heavily relies on the accuracy of the initial transformation matrix or requires additional camera intrinsic parameter information for supervision. In this paper, we proposed a sub-pixel position error estimation network (SPEN) for multi-modal image matching. The proposed method includes three modules: multi-scale feature extraction, pixel-level reliable feature matching, and sub-pixel position error estimation. The multi-scale feature extraction module integrates an adaptive encoder for modality changes and a multi-scale feature fusion block for geometric distortions to extract robust descriptors. The pixel-level reliable feature matching module designs a detection-description coupling detector, which detects reliable keypoints by emphasizing both keypoint repeatability and matchability, thus improving the overall matching performance. The sub-pixel position error estimation module utilizes the reprojection error function with sub-pixel accuracy to supervise the regression of position errors, improving the alignment between point pairs without depending on the accuracy of the initial transformation matrix. Furthermore, this module only requires the transformation matrix generated by random affine adaptation and data augmentation to calculate the reprojection error rather than additional camera intrinsic, making it more versatile and suitable for a broader range of applications. Experimental results demonstrate the superiority of the proposed method over the state-of-the-art methods on three multi-modal image datasets. Additionally, the ablation study highlights the effectiveness of the proposed components. Our implementation will be available at https://github.com/huhulike/SPEN.

The rapid progression of remote sensing technologies has positioned the effective exploitation of multi-source data—e.g., hyperspectral imaging (HSI) and light detection and ranging (LiDAR) data—for land cover classification as a pivotal research domain. However, existing methods fail to fully consider the interactions between different source features. Additionally, how to fully exploit discriminative structural information in the frequency domain remains a critical challenge. To this end, a novel multi-domain adaptive fusion network (MDAF-Net) is proposed for the joint classification of HSI and LiDAR data. MDAF-Net integrates multi-scale feature extraction, adaptive spatial-channel interaction, and frequency-aware fusion, which achieves comprehensive multi-domain representation and fusion across the spatial, spectral, and frequency domains. Specifically, a multi-scale module is first introduced to extract hierarchical features from the input data. Then, to enhance adaptive interaction between spatial and channel dimensions, an adaptive spatial-channel exchange module (ASCEM) is proposed to capture intra-feature dependencies and cross-modal correlations. Finally, a frequency-aware fusion module (FAFM) is proposed to fully leverage frequency domain information. This module adaptively adjusts frequency components, enhancing structural feature representations. Extensive experiments on three widely used datasets demonstrate that the proposed method outperforms existing state-of-the-art methods.

Remote sensing image change captioning is an important image interpretation technique that automatically generates captions describing the visual changes in multitemporal remote sensing images. However, the visual changes present in multitemporal images can be classified as foreground changes, which are captured in captions, and background changes, which interfere with traditional methods and complicate the effective capture of foreground changes. This ultimately limits the overall performance of the model. To address this issue, this study introduces change detection for remote sensing image change captioning (CD4C). Specifically, a change detection module generates binary masks that contain relevant visual change information from multitemporal images. Subsequently, based on whether changes are detected, samples are classified and processed through the C-Stream and N-Stream of the multitemporal difference feature fusion (MDF) module to extract visual change features. The C-Stream leverages the visual change information provided by the mask to enhance the ability of CD4C to capture foreground visual change features at both the image and feature levels. The N-Stream incorporates a pseudofeature generation module designed to mitigate the interference caused by poor change detection results. Finally, the caption generation module interprets the visual change features extracted by the MDF to produce accurate textual descriptions. Experiments on the LEVIR-CC and Dubai-CC datasets demonstrate that the proposed method outperforms other approaches.

Knowledge distillation (KD) improves the performance of a compact student network by transferring learned knowledge from a cumbersome teacher network. In the existing approaches, the multiscale feature knowledge is transferred via densely connected paths, which increases the optimization difficulty. Moreover, correlations among the labels are neglected despite their capability to enhance the intraclass similarity of samples. To solve these issues, we propose cascade fusion and correlation enhancement for KD (CC-KD). The multiscale feature knowledge is transferred via much simpler paths, which are constructed by fusing features of different scales with cross-scale attention (CSA) in a cascade manner, thereby reducing the optimization difficulty. On the other hand, the relational knowledge of teacher logits is further enhanced by correlations of the corresponding labels, so that the student can produce more similar logits for the samples in the same category. Extensive experimental results on five public datasets (i.e., CIFAR100/10, ImageNet, RAF-DB, and FERPlus) indicate superior performance of the proposed method over several state-of-the-arts (SOTAs). More specifically, our method obtains an accuracy of 71.70% on ImageNet and achieves a new record of 90.20% on RAF-DB with fewer calculations and parameters.

Image matching is a critical task in computer vision research, focusing on aligning two or more images with similar features. Feature detection and description constitute the core of image matching. Handcrafted detectors are capable of obtaining distinctive points but these points may not be repeatable on the image pairs especially those with dramatic appearance changes. On the contrary, the learned detectors can extract a large number of repeatable points but many of them tend to be ambiguous points with low distinctiveness. Moreover, in the scenarios of dramatic appearance change, commonly used contrast or triplet loss in the training of descriptors employ the hard negative mining strategy, which may obtain overly challenging negative samples by global sampling, resulting in sluggish convergence or even overfitting. Those learned descriptors may not guarantee that the corresponding points enjoy larger similarities than unmatched ones, leading to inaccurate matches. To address those issues, we propose a hierarchically learned detector and descriptor (HLDD) for robust image matching, which contains three modules: a handcrafted-learned detector, a hierarchically learned descriptor, and a coarse-to-fine matching strategy. The handcrafted-learned detector integrates the advantages of handcrafted and learned detectors. It extracts distinctive feature points from a learned repeatability map robust to image changes and eliminates the ambiguous ones according to a learned distinctiveness map. The descriptor is trained by a proposed hierarchical triplet loss, which employs a dual window strategy. It can obtain the hardest negative samples in local windows, which are comparatively easier over global sampling, ensuring the effective training of descriptors. The coarse-to-fine matching strategy performs global and local mutual nearest neighbor matching on the coarse and fine descriptor maps respectively to improve the matching accuracy progressively. By comparing with other matching methods, experimental results demonstrate the superiority of the proposed method in the task of image matching, homography estimation, visual localization, and relative pose estimation. Moreover, ablation studies illustrate the effectiveness of the three proposed modules.

Nowadays, power line surveillance widely utilizes infrared images collected by drones for instance segmentation (IS) to determine whether the power equipment is abnormal, such as overheating. With the development of deep learning, state-of-the-art IS methods can be effectively applied to infrared images of electrical scenarios. However, the low spatial resolution and limited data scale of power line infrared images pose challenges for these methods. In this article, we make the following contributions: 1) we present a real-world large-scale dataset, CSG, which includes 11715 power line infrared images with a spatial resolution of $640 \times 512$ , and a total of 45145 instances. The CSG dataset covers 7 foreground objects coupled with different backgrounds, and has various scenarios from 295 geosites, providing a benchmark for evaluating the performance of IS methods and 2) we design a multitask learning (MTL) framework, PowerNet, where super-resolution (SR) and IS are unified into a collaborative network. SR allows the preservation of high-resolution (HR) feature representations, while IS incorporates them through a knowledge transfer pipeline (KTP). This enhances the discriminative ability of objects, particularly those that are small and complex, leading to more accurate predictions under the condition of low-resolution (LR) images. The pipeline measures task affinity for knowledge transfer and is effectively supervised by multitask loss. Extensive experiments reveal that the proposed framework achieves remarkable performance on the CSG dataset.

Most of the existing methods for video understanding primarily focus on videos only lasting tens of seconds, with limited exploration of techniques for handling long videos. The increased number of frames in long videos poses two main challenges: difficulty in locating key information and performing long-range reasoning. Thus, we propose DrVideo, a document-retrieval-based system designed for long video understanding. Our key idea is to convert the long-video understanding problem into a long-document understanding task so as to effectively leverage the power of large language models. Specifically, DrVideo first transforms a long video into a coarse text-based long document to initially retrieve key frames and then updates the documents with the augmented key frame information. It then employs an agent-based iterative loop to continuously search for missing information and augment the document until sufficient question-related information is gathered for making the final predictions in a chain-of-thought manner. Extensive experiments on long video benchmarks confirm the effectiveness of our method. DrVideo significantly outperforms existing LLM-based state-of-the-art methods on EgoSchema benchmark (3 minutes), MovieChat-1K benchmark (10 minutes), and the long split of Video-MME benchmark (average of 44 minutes). Code is available at https://github.com/Upper9527/DrVideo.