Optical-SAR fusion object detection has drawn great attention for its ability to combine the rich visual features of optical images with the all-weather, all-day imaging capability of SAR, enabling robust detection performance under adverse weather conditions. However, cloud occlusion causes severe degradation in optical images, while the weak texture of SAR images leads to missed detections, which poses significant challenges to accurate object detection. To address this issue, we propose a novel cross-modal feature calibration and fusion network (CFCFNet) for robust object detection under cloud occlusion. Specifically, we propose an R2U-Net-based cloud removal module to enable end-to-end cloud removal at a low computational cost. To enhance feature representations and mitigate cross-modal discrepancies, a cross-modal feature calibration module is introduced for channel-wise and spatial feature alignment. Finally, to fully exploit the complementary information across modalities, we design a convolutional attention-based dynamic fusion module that adaptively integrates global context and local details to enhance the discriminative capability of fused features. Besides, we construct the OPTSAR-Cloud dataset, comprising five sets of 3300 optical-SAR image pairs under varying cloud thickness, enabling evaluation under complex atmospheric environments. Extensive experiments on the OPTSAR-Cloud dataset demonstrate that CFCFNet achieves state-of-the-art performance, with an absolute gain of 2.7% in mAP50 over previous best approaches. Ablation studies under thick cloud conditions further show that CFCFNet improves detection accuracy by 5.6%, increasing mAP50 from 60.8% to 66.4%.

Multimodal remote sensing object detection (MM-RSOD) holds great promise for around-the-clock applications. However, it faces challenges in effectively extracting complementary features due to the modality inconsistency and redundancy. Inconsistency can lead to semantic-spatial misalignment, while redundancy introduces uncertainty that is specific to each modality. To overcome these challenges and enhance complementarity exploration and exploitation, this article proposes a dual-dynamic cross-modal interaction network (DDCINet), a novel framework comprising two key modules: a dual-dynamic cross-modal interaction (DDCI) module and a dynamic feature fusion (DFF) module. The DDCI module simultaneously addresses both modality inconsistency and redundancy by employing a collaborative design of channel-gated spatial cross-attention (CSCA) and cross-modal dynamic filters (CMDFs) on evenly segmented multimodal features. The CSCA component enhances the semantic-spatial correlation between modalities by identifying the most relevant channel-spatial features through cross-attention, addressing modality inconsistency. In parallel, the CMDF component achieves cross-modal context interaction through static convolution and further generates dynamic spatial-variant kernels to filter out irrelevant information between modalities, addressing modality redundancy. Following the improved feature extraction, the DFF module dynamically adjusts interchannel dependencies guided by modal-specific global context to fuse features, achieving better complementarity exploitation. Extensive experiments conducted on three MM-RSOD datasets confirm the superiority and generalizability of the DDCINet framework. Notably, our DDCINet, based on the RoI Transformer benchmark and ResNet50 backbone, achieves 78.4% mAP50 on the DroneVehicle test set and outperforms state-of-the-art (SOTA) methods by large margins.

In numerous all-weather applications, multi-modal pedestrian detection has been widely adopted, notably in autonomous driving and video surveillance. Exploiting the complementary strengths of different modalities is one of the most effective ways to enhance detection performance. However, misalignment in the spatial domain and variability in modality reliability can hinder robust cross-modal feature extraction, thereby limiting the potential of multi-modal pedestrian detection.To address these issues, this paper proposes a cross-modal object detection method based on unregistered optical and infrared images, termed the Frequency-Domain-Based Multispectral Pedestrian Detection Network (FD-MPedNet). Specifically, we design a Multi-Scale Channel-Spatial (MSCS) module that leverages deformable convolutions and multi-scale convolution operations to effectively capture edge information and extract features across different spatial scales, while preserving channel priors to avoid disregarding their inherent feature distributions. Additionally, we introduce an Adaptive Frequency Domain Feature Fusion (AFDFF) module, wherein the frequency bands of different modalities are decomposed to fuse low-frequency components for global information exchange, while retaining the original high-frequency features to enhance the model’ s sensitivity to object pixels. Experimental results on the challenging KAIST dataset demonstrate that the proposed FD-MPedNet achieves outstanding performance.

Passive non-line-of-sight (NLOS) reconstruction has received considerable success in diverse fields. However, the existing reconstruction methods ignore that complex scenes attenuate object-related information and view object-related information and noise in measured images as equivalent, yielding low-quality recovery. We propose an attention-based encoder–decoder (AED) network to tackle this problem. Specifically, we introduce an attention in the attention (A2B) module that can prune the attention layers to help the network focus on the object-related information in the measured images. In addition, we establish several datasets in complex scenes, including varying ambient light conditions and parameter settings of reconstruction systems, as well as complex hidden objects, to verify the generalization of our method. Experiments on our constructed datasets demonstrate that our methods achieve better recovery performance than existing methods, with more robustness to complex scenes.

Land use classification using optical and Synthetic Aperture Radar (SAR) images is a crucial task in remote sensing image interpretation. Recently, deep multi-modal fusion models have significantly enhanced land use classification by integrating multi-source data. However, existing approaches solely rely on simple fusion methods to leverage the complementary information from each modality, disregarding the intermodal correlation during the feature extraction process, which leads to inadequate integration of the complementary information. In this paper, we propose FASONet, a novel multi-modal fusion network consisting of two key modules that tackle this challenge from different perspectives. Firstly, the feature alignment module (FAM) facilitates cross-modal learning by aligning high-level features from both modalities, thereby enhancing the feature representation for each modality. Secondly, we introduce the multi-modal squeeze and excitation fusion module (MSEM) to adaptively fuse discriminative features by weighting each modality and removing irrelevant parts. Our experimental results on the WHU-OPT-SAR dataset demonstrate the superiority of FASONet over other fusion-based methods, exhibiting a remarkable 5.1% improvement in MIoU compared to the state-of-the-art MCANet method.

Multispectral pedestrian detection can provide accurate and reliable results from color-thermal modalities and has drawn much attention. However, how to effectively capture and leverage complementary information from multiple modalities for superior performance is still a core issue. This paper presents a Cross-Modal Attentive Recalibration and Dynamic Fusion Network (CMRF-Net) to adaptively recalibrate and dynamically fuse multi-modal features from multiple perspectives. CMRF-Net consists of a Cross-modal Attentive Feature Recalibration (CAFR) module and a Multi-Modal Dynamic Feature Fusion (MDFF) module in each feature extraction stage. The CAFR module recalibrates features by fully leveraging local and global complementary information in spatial- and channel-wise dimensions, leading to better cross-modal feature alignment and extraction. The MDFF module adopts dynamically learned convolutions to further exploit complementary information in kernel space, enabling more efficient multi-modal feature aggregation. Extensive experiments are conducted on three multispectral datasets to show the effectiveness and generalization of the proposed method and the state-of-the-art detection performance. Specifically, CMRF-Net can achieve 2.3% mAP gains over the baseline on FLIR dataset.

Unlike able-bodied persons, it is difficult for visually impaired people, especially those in the educational age, to build a full perception of the world due to the lack of normal vision. The rapid development of AI and sensing technologies has provided new solutions to visually impaired assistance. However, to our knowledge, most previous studies focused on obstacle avoidance and environmental perception but paid less attention to educational assistance for visually impaired people. In this paper, we propose AviPer, a system that aims to assist visually impaired people to perceive the world via creating a continuous, immersive, and educational assisting pattern. Equipped with a self-developed flexible tactile glove and a webcam, AviPer can simultaneously predict the grasping object and provide voice feedback using the vision-tactile fusion classification model, when a visually impaired people is perceiving the object with his gloved hand. To achieve accurate multimodal classification, we creatively embed three attention mechanisms, namely temporal, channel-wise, and spatial attention in the model. Experimental results show that AviPer can achieve an accuracy of 99.75% in classification of 10 daily objects. We evaluated the system in a variety of extreme cases, which verified its robustness and demonstrated the necessity of visual and tactile modal fusion. We also conducted tests in the actual use scene and proved the usability and user-friendliness of the system. We opensourced the code and self-collected datasets in the hope of promoting research development and bringing changes to the lives of visually impaired people.

Due to the vast geometric and radiometric differences between SAR and optical images, SAR-optical image matching remains an intractable challenge. Despite the fact that the deep learning-based matching model has achieved great success, SAR feature embedding ability is not fully explored yet because of the lack of well-designed pre-training techniques. In this paper, we propose to employ the self-supervised learning method in the SAR-optical matching framework, in order to serve as a pre-training strategy for improving the representation learning ability of SAR images as well as optical images. We first use a state-of-the-art self-supervised learning method, Momentum Contrast (MoCo), to pre-train an optical feature encoder and an SAR feature encoder separately. Then, the pre-trained encoders are transferred to an advanced common representation learning model, Bridge Neural Network (BNN), to project the SAR and optical images into a more distinguishable common feature representation subspace, which leads to a high multi-modal image matching result. Experimental results on three SAR-optical matching benchmark datasets show that our proposed MoCo pre-training method achieves a high matching accuracy up to 0.873 even for the complex QXS-SAROPT SAR-optical matching dataset. BNN pre-trained with MoCo outperforms BNN with the most commonly used ImageNet pre-training, and achieves at most 4.4% gains in matching accuracy.

Airplane detection in synthetic aperture radar (SAR) images has drawn much attention owing to the success of deep learning methods. However, the development of fine-grained airplane detection in SAR images is still in a dilemma due to the small interclass variance and the large intraclass variance in complex scenes with strong interference from the background. In addition, the class imbalance problem in multiclass fine-grained airplane recognition also significantly limits the direct application of general deep-learning-based airplane detectors. This article proposes two effective methods to tackle the above two problems, respectively. First, we propose a sparse attention-guided fine-grained pyramid module to simultaneously sample discriminative local features scattered in multiscale layers and adaptively aggregate them with fine-grained attention to better classify subordinate-level airplanes with multiple scales. Second, a simple class-balanced copy-paste data augmentation strategy, which randomly copies an airplane of one category and pastes it onto an image according to the classwise probability, is proposed for class balance. Finally, extensive experiments on one public dataset and three representative deep-learning-based detection benchmarks are conducted to show the effectiveness and generalization of the two proposed methods. The combination of these two methods based on the cascade R-CNN benchmark also won the fifth place in fine-grained airplane detection in SAR images in the 2021 GaoFen Challenge.

Zero-shot knowledge graph (KG) has gained much research attention in recent years. Due to its excellent performance in approximating data distribution, generative adversarial network (GAN) has been used in zero-shot learning for KG completion. However, existing works on GAN-based zero-shot KG completion all use traditional simple architecture without randomness in generator, which greatly limits the ability of GAN mining knowledge on complex datasets. Moreover, the discriminator not only needs to distinguish true data from generated data but also needs to classify generated data correctly at the same time, which affects the optimization process of the generator. In this work, we propose a novel zero-shot KG framework based on stochastic and dual adversarial GAN (SDA) to better mine the association between semantic information and extracted features. Specifically, we introduce a stochastic generator and an additional classifier to improve the model's ability of approximating features and classifying unseen tasks. The experiments on NELL-ZS and Wiki-ZS datasets show that the proposed SDA outperforms the classic methods in zero-shot KG completion task. In particular, the proposed SDA receives a 0.6% and 0.7% increase on mean reciprocal ranking (MRR) for NELL-ZS and Wiki-ZS datasets, respectively.

Multispectral pedestrian detection provides more accurate and reliable detection results by leveraging complementary information from color-thermal modalities and has drawn much attention in the open world. Much progress has been made in the feature-level-based detection methods which aim to effectively fuse the multispectral features extracted by the convolution neural networks. However, existing methods mainly focus on the information integration between the same-level feature maps and ignore the complementary local features scattered in multi-scale layers. In this paper, we introduce an Attention-guided multi-Modal and multi-Scale Fusion (AMSF) module to simultaneously sample complementary local features scattered in multi-modal and multi-scale layers, and adaptively aggregate them with fine-grained attention to fully exploit different modalities for better multi-scale detection results. Extensive experiments are conducted on three multispectral datasets and three representative deep-learning-based detection benchmarks to show the effectiveness and generalization of the proposed method, and the state-of-the-art detection performance.

Few-shot classification is a challenging task which aims to formulate the ability of humans to learn concepts from limited prior data and has drawn considerable attention in machine learning. Recent progress in few-shot classification has featured meta-learning, in which a parameterized model for a learning algorithm is defined and trained to learn the ability of handling classification tasks on extremely large or infinite episodes representing different classification task, each with a small labeled support set and its corresponding query set. In this work, we advance this few-shot classification paradigm by formulating it as a supervised classification learning problem. We further propose multi-episode and cross-way training techniques, which respectively correspond to the minibatch and pretraining in classification problems. Experimental results on a state-of-the-art few-shot classification method (prototypical networks) demonstrate that both the proposed training strategies can highly accelerate the training process without accuracy loss for varying few-shot classification problems on Omniglot and miniImageNet.

Non-line-of-sight (NLOS) imaging is attractive for its potential applications in autonomous vehicles, robotic vision, and biomedical imaging. NLOS imaging can be realized through reconstruction or recognition. Recognition is preferred in some practical scenarios because it can classify hidden objects directly and quickly. Current NLOS recognition is mostly realized by exploiting active laser illumination. However, passive NLOS recognition, which is essential for its simplified hardware system and good stealthiness, has not been explored. Here, we use a passive imaging setting that consists of a standard digital camera and an occluder to achieve a NLOS recognition system by deep learning. The proposed passive NLOS recognition system demonstrates high accuracy with the datasets of handwritten digits, hand gestures, human postures, and fashion products (81.58 % to 98.26%) using less than 1 second per image in a dark room. Beyond, good performance can be maintained under more complex lighting conditions and practical tests. Moreover, we conversely conduct white-box attacks on the NLOS recognition algorithm to study its security. An attack success rate of approximately 36% is achieved at a relatively low cost, which demonstrates that the existing passive NLOS recognition remains somewhat vulnerable to small perturbations.

The current interpretation technology of remote sensing images is mainly focused on single-modal data, which cannot fully utilize the complementary and correlated information of multimodal data with heterogeneous characteristics, especially for synthetic aperture radar (SAR) data and optical imagery. To solve this problem, we propose a bridge neural network- (BNN-) based optical-SAR image joint intelligent interpretation framework, optimizing the feature correlation between optical and SAR images through optical-SAR matching tasks. It adopts BNN to effectively improve the capability of common feature extraction of optical and SAR images and thus improving the accuracy and application scenarios of specific intelligent interpretation tasks for optical-SAR/SAR/optical images. Specifically, BNN projects optical and SAR images into a common feature space and mines their correlation through pair matching. Further, to deeply exploit the correlation between optical and SAR images and ensure the great representation learning ability of BNN, we build the QXS-SAROPT dataset containing 20,000 pairs of perfectly aligned optical-SAR image patches with diverse scenes of high resolutions. Experimental results on optical-to-SAR crossmodal object detection demonstrate the effectiveness and superiority of our framework. In particular, based on the QXS-SAROPT dataset, our framework can achieve up to 96% high accuracy on four benchmark SAR ship detection datasets.

Deep learning methods have made significant progress in ship detection in synthetic aperture radar (SAR) images. The pretraining technique is usually adopted to support deep neural networks-based SAR ship detectors due to the scarce labeled SAR images. However, directly leveraging ImageNet pretraining is hard to obtain a good ship detector because of different imaging perspectives and geometry. In this article, to resolve the problem of inconsistent imaging perspectives between ImageNet and earth observations, we propose an optical ship detector (OSD) pretraining technique to transfer the characteristics of ships in earth observations to SAR images from a large-scale aerial image dataset. On the other hand, to handle the problem of different imaging geometry between optical and SAR images, we propose an optical-SAR matching (OSM) pretraining technique, which transfers plentiful texture features from optical images to SAR images by common representation learning on the OSM task. Finally, observing that the OSD pretraining-based SSD has a better recall on sea area while the OSM pretraining-based SSD can reduce false alarms on land area, we combine the predictions of the two detectors through weighted boxes fusion to further improve detection results. Extensive experiments on four SAR ship detection datasets and three representative convolutional neural network-based detection benchmarks are conducted to show the effectiveness and complementarity of the two proposed detectors, and the state-of-the-art performance of the combination of the two detectors. The proposed method won the sixth place of ship detection in SAR images in the 2020 Gaofen challenge.

For the space-based remote sensing system, onboard intelligent processing based on deep learning has become an inevitable trend. To adapt to the dynamic changes of the observation scenes, there is an urgent need to perform distributed deep learning onboard to fully utilize the plentiful real-time sensing data of multiple satellites from a smart constellation. However, the network bandwidth of the smart constellation is very limited. Therefore, it is of great significance to carry out distributed training research in a low-bandwidth environment. This paper proposes a Randomized Decentralized Parallel Stochastic Gradient Descent (RD-PSGD) method for distributed training in a low-bandwidth network. To reduce the communication cost, each node in RD-PSGD just randomly transfers part of the information of the local intelligent model to its neighborhood. We further speed up the algorithm by optimizing the programming of random index generation and parameter extraction. For the first time, we theoretically analyze the convergence property of the proposed RD-PSGD and validate the advantage of this method by simulation experiments on various distributed training tasks for image classification on different benchmark datasets and deep learning network architectures. The results show that RD-PSGD can effectively save the time and bandwidth cost of distributed training and reduce the complexity of parameter selection compared with the TopK-based method. The method proposed in this paper provides a new perspective for the study of onboard intelligent processing, especially for online learning on a smart satellite constellation.

Deep learning techniques have made an increasing impact on the field of remote sensing. However, deep neural networks based fusion of multimodal data from different remote sensors with heterogenous characteristics has not been fully explored, due to the lack of availability of big amounts of perfectly aligned multi-sensor image data with diverse scenes of high resolutions, especially for synthetic aperture radar (SAR) data and optical imagery. To promote the development of deep learning based SAR-optical fusion approaches, we release the QXS-SAROPT dataset, which contains 20,000 pairs of SAR-optical image patches. We obtain the SAR patches from SAR satellite GaoFen-3 images and the optical patches from Google Earth images. These images cover three port cities: San Diego, Shanghai and Qingdao. Here, we present a detailed introduction of the construction of the dataset, and show its two representative exemplary applications, namely SAR-optical image matching and SAR ship detection boosted by cross-modal information from optical images. As a large open SAR-optical dataset with multiple scenes of a high resolution, we believe QXS-SAROPT will be of potential value for further research in SAR-optical data fusion technology based on deep learning.

Metric-learning-based methods, which attempt to learn a deep embedding space on extremely large episodes, have been successfully applied to few-shot classification problems. In this paper, we propose the adoption of large margin nearest center (LMNC) loss during episodic training to enhance metric-learning-based few-shot classification methods. Loss functions (such as cross-entropy and mean square error) commonly used in episodic training strive to achieve the strict goal that differently labeled examples in the embedding space are separated by an infinite distance. However, the learned embedding space cannot guarantee that this goal will be achieved for every episode sampled from a large number of classes. Instead of an infinite distance, LMNC loss requires only that differently labeled examples be separated by a large margin, which can well relax the strict constraint of the traditional loss functions, easily leading to a discriminative embedding space. Moreover, considering the multilevel similarity between various classes, we alleviate the constraint of a fixed large margin and extend LMNC loss to weighted LMNC (WLMNC) loss, which can effectively take advantage of interclass information, achieving a more separable embedding space with adaptive interclass margins. Experiments on state-of-the-art benchmarks demonstrate that the adoption of LMNC and WLMNC losses can strongly improve the embedding learning performance and classification accuracy of metric-based few-shot classification methods for various few-shot scenarios. In particular, LMNC and WLMNC losses can obtain 1.86% and 2.46% gains in prototypical network on miniImageNet for 5-way 1-shot scenario, respectively.

Deep neural networks have dramatically achieved great success on a variety of challenging tasks. However, most successful DNNs have an extremely complex structure, leading to extensive research on model compression. As a significant area of progress in model compression, traditional gradual pruning approaches involve an iterative prune-retrain procedure and may suffer from two critical issues: local importance judgment, where the pruned weights are merely unimportant in the current model; and an irretrievable pruning process, where the pruned weights have no chance to come back. Addressing these two issues, this paper proposes the Drop Pruning approach, which leverages stochastic optimization in the pruning process by introducing a drop strategy at each pruning step, namely, drop away, which stochastically deletes some unimportant weights, and drop back, which stochastically recovers some pruned weights. The suitable choice of drop probabilities decreases the model size during the pruning process and helps it flow to the target sparsity. Compared to the Bayesian approaches that stochastically train a compact model for pruning, we directly aim at stochastic gradual pruning. We provide a detailed analysis showing that the drop away and drop back approaches have individual contributions. Moreover, Drop Pruning can achieve competitive compression performance and accuracy on many benchmark tasks compared with state-of-the-art weights pruning and Bayesian training approaches.

Transfer learning aims to learn robust classifiers for the target domain by leveraging knowledge from a source domain. Since the source and the target domains are usually from different distributions, existing methods mainly focus on adapting the cross-domain marginal or conditional distributions. However, in real applications, the marginal and conditional distributions usually have different contributions to the domain discrepancy. Existing methods fail to quantitatively evaluate the different importance of these two distributions, which will result in unsatisfactory transfer performance. In this article, we propose a novel concept called Dynamic Distribution Adaptation (DDA), which is capable of quantitatively evaluating the relative importance of each distribution. DDA can be easily incorporated into the framework of structural risk minimization to solve transfer learning problems. On the basis of DDA, we propose two novel learning algorithms: (1) Manifold Dynamic Distribution Adaptation (MDDA) for traditional transfer learning, and (2) Dynamic Distribution Adaptation Network (DDAN) for deep transfer learning. Extensive experiments demonstrate that MDDA and DDAN significantly improve the transfer learning performance and set up a strong baseline over the latest deep and adversarial methods on digits recognition, sentiment analysis, and image classification. More importantly, it is shown that marginal and conditional distributions have different contributions to the domain divergence, and our DDA is able to provide good quantitative evaluation of their relative importance, which leads to better performance. We believe this observation can be helpful for future research in transfer learning.