With the rapid growth of the Internet, cross-modal retrieval (CMR) has attracted increasing attention for its ability to bridge heterogeneous modalities. Most existing studies assume that data annotations are completely accurate, but this rarely holds in practice since both human non-expert and machine annotations often introduce noisy labels. To mitigate the effect of label noise, various robust learning approaches have been proposed with promising results. However, these methods are generally developed under a closed-set assumption, where noisy samples remain within the known label space. In real-world scenarios, noisy samples may originate from unseen categories, referred to as open-set noisy labels, under which existing methods often fail. To address this challenge, we propose a novel CMR framework named Optimal Transport Filtering with Open-Set Noisy Labels (OTOS). Specifically, a Discriminative Reinforcement Learning (DRL) module is introduced to enhance instance-level discrimination and reduce multimodal heterogeneity, while an Optimal Transport Filtering (OTF) module leverages geometric distances to identify clean samples, closed-set noise, and open-set noise effectively. Furthermore, specialized learning strategies are designed for different instance types to fully exploit the information contained in both closed-set and open-set data. Extensive experiments on three benchmark datasets demonstrate the superior performance of OTOS, particularly in handling multimodal data with open-set noisy labels.

Balancing convergence, diversity, and feasibility is a critical challenge in constrained multi-objective optimization, particularly when dealing with discontinuous or degenerate Pareto fronts (PFs). Constrained multi-objective evolutionary algorithms (CMOEAs) are commonly employed to tackle such problems, yet most of them prioritize convergence and feasibility while overlooking diversity. To address this gap, we propose a novel two-stage hybrid tri-population CMOEA (TS-HC3) that enhances both exploration and exploitation to achieve well-converged, diverse, and feasible solutions. In the first stage, a dual-population exploration mechanism is presented to promotes broad exploration and accelerate the discovery of promising feasible regions. In the second stage, a tri-population exploitation mechanism is introduced, incorporating a novel subregion-aware environmental selection strategy that adaptively preserves solution distribution and uniformity across the PF. A two-level multi-armed bandit strategy is proposed to dynamically select the most suitable parent population and variation operator based on real-time evolutionary feedback. Finally, a differential evolution based flexible genetic operator is designed to enhance exploration in sparse feasible regions. Extensive experiments on widely used benchmarks demonstrate that TS-HC3 significantly outperforms state-of-the-art CMOEAs, particularly on problems with discontinuous or degenerate PFs. The effectiveness of TS-HC3 is further validated on a multi-unmanned surface vehicle collaborative path planning problem, highlighting its practical applicability in complex real-world scenarios. Moreover, an ablation study is conducted to validate the contribution of each individual component within TS-HC3.

Different from traditional object detection, pure vision is often not enough to infrared small target detection (ISTD), due to the small target size and weak background contrast. For promoting performance, more target representations are often needed. Currently, motion representations have been proved to be one of the most potential feature patterns for infrared small targets. Besides vision features, existing methods have an obvious weakness that they could only capture coarse motion representations from the temporal domain. By vision features, fine motion representations could often be more effective to enhance detection performance. To overcome this weakness, and inspired by prevalent vision-language models (VLMs), the first vision-language framework with motion prior knowledge learning (MoPKL) was proposed in our previous work. To further extend it, we repropose an improved version, i.e., iMoPKL. Breaking through traditional pure-vision modality, it utilizes the homogeneous language descriptions, specially formatted for moving targets, to directionally guide vision channels to learn the motion prior knowledge of targets. In detail, it learns the distribution of target motion reconstruction corresponding to the language description as a type of prior knowledge. With the facilitation of language-driven motion alignment, the motion of infrared small targets could be further refined by motion-relation learning, to generate more fine motion representations. The extensive experiments on ITSDT-15K, DAUB-R, and IRDST-H show that our improvement version is effective. It could often obviously outperform the other methods, including our original MoPKL. Our source codes are available at https://github.com/UESTC-nnLab/MoPKL

Facial micro-expressions (ME) are fast, minute movements of the facial muscles that communicate emotions and intentions that are concealed. Due to limited datasets, ephemeral occurrence, subtle movement of facial muscles, subjective classification, noise and background, and ethical quandaries, it is difficult to identify ME from videos. This study proposes the deep model DITRAGS-STNet and the FHOG-DITRAGS-STNet comprising the dynamic texture handcrafted feature descriptor HOG-TOP and the optical flow feature descriptor Bi-WOOF, and the smaller size DITRAGS-STNet deep model to classify ME videos. Both models use deep channels that integrate dense CNN blocks, customized transition blocks with factorized and residual inception, and GRUs with soft temporal attention to create high-level features. The random forest ensemble classifier is used to categorize the feature vectors into three classes: negative, positive, and surprise. FHOG-DITRAGS-STNet fuses the complementary capabilities of three feature sources to improve classification performance. Two composite datasets are constructed: MASPOWER, which mitigates overfitting through domain adaptation, and CASSMEW-MICRO, which enhances ME recognition across diverse samples while increasing sample size. These datasets are created using five benchmark macro-expression datasets and six benchmark ME datasets, respectively. Extensive cross-validations demonstrate that both models are superior in terms of efficacy and generality.

RGB-Event object tracking has demonstrated significant potential in addressing challenging scenarios such as motion blur and high dynamic range, with its success largely hinging on the efficient fusion of visual information from both modalities. Existing Transformer-based methods achieve strong performance in multimodal learning but are hindered by high memory and computational costs, limiting practical deployment. Recently, State Space Model (SSM) has emerged as an effective mechanism for global token interactions with linear computational complexity. However, their inherent causal processing disrupts spatial structures in images, limiting their suitability for non-causal vision tasks. To address this issue, this paper proposes a novel RGB-Event tracking framework based on a non-causal format of State Space Duality (SSD). The framework comprises a backbone network for joint feature extraction and relation modeling, along with a fusion module to facilitate complementary integration of RGB and event modalities, achieving both efficiency and high performance. Extensive experiments on the FE108 and VisEvent datasets validate the effectiveness of the proposed method, showing significant improvements in tracking accuracy compared to state-of-the-art SSM-based trackers.

As an important research branch of infrared small target detection, dense target detection (e.g., drone swarm detection) has always been a topic worth exploring. Currently, existing datasets cover only one or several (sparse) targets, with almost no dataset available for the research on dense small target detection. To advance this kind of search, for the first time, we synthesize two special dense moving target datasets (DMIST-60 and DMIST-100) on DAUB data. They both contain far more than 50 infrared small targets per frame. In the meantime, for evaluating our new datasets and flourishing detection methodology research, we propose a linking-aware sliced network (LASNet) as the baseline of our datasets. It mainly consists of visual feature extraction, motion feature extraction and motion-affinity fusion. The comprehensive experiments on our synthesized datasets confirm: i) both datasets are practical and effective for dense moving infrared small target detection and ii) proposed LASNet could always obviously outperform other compared methods in both sparse and dense target scenarios.

Imbalanced data problem is a classic topic in artificial intelligence. Neural network approaches to solve this problem mostly rely on resampling or reweighting strategies. However, these methods severely suffer from the learning bias in most cases when the empirical representation of known samples is insufficient. One-class learning can provide an ideal classification property to alleviate this critical issue. However, extending one-class learning to imbalanced data presents problems of hypersphere collapse, ambiguous interclass relations, and compact representations. In this paper, a new one-class learning paradigm is proposed for binary imbalanced data classification. Specifically, a neural network is employed to map known samples to a specified attribute space to solve the problems of hypersphere collapse and ambiguous interclass relations. Then, to alleviate the compact representation problem, a dynamic information potential energy is developed to disperse the mapped majority samples to fill the specified region as much as possible. The proposed method is validated on 34 imbalanced datasets with imbalanced ratios ranging from 16.90 to 100.14. The test results show that the proposed method achieves the best performance on more than half of the test datasets.

Neural networks handling data imbalance heavily rely on resampling or reweighting strategies. However, existing resampling and reweighting approaches mainly focus on rebalancing known data, which ignore the essence of the data imbalance problem, namely, the problem of insufficient empirical representation of the minority class caused by the small number of samples. Therefore, we propose a new solution for neural networks classifying imbalanced data by sampling absent minority class samples. Specifically, an improved Metropolis Hasting (IMH) algorithm is developed to sample absent minority class samples by collecting samples rejected by the majority class approximation process. The sampled absent minority samples are then provided to neural networks to address the data imbalance problem. For IMH, in order to accelerate the sampling process and reduce the vague class definition of the sampled minority class samples, line segment transition kernel and class probability constraint are proposed. For neural networks, two boundary shifting strategies are supported to operate on different application modes of sampled absent minority class samples. In experiments, the proposed method is validated on 34 imbalanced datasets. Comparable AUC, G-MEAN, and MACC results are achieved. These results demonstrate the effectiveness of sampling absent minority class samples for neural networks solving the imbalanced data problem.

Facial Micro-Expression (ME) is one of the pre-dominating non-verbal clues to demystify the true emotional states that people try to conceal cautiously. But emotion recognition from spontaneous ME images limits the high accuracy due to short duration and low intensity, lack of sufficient samples and consistencies among publicly available ME datasets. In this study, we have proposed two highly effective, lightweight, and generalized single-channel DLRRF-MER, and multi-channel DLH-3C-FUSION fusion models inspired by deep dense convolutional models and texture-based feature descriptors Local Binary Pattern (LBP) and Histogram of Oriented Gradients (HOG) to recognize ME from apex frame by constructing a composite dataset from five publicly available ME datasets CASME, CASMEII, CAS(ME) 2 , SAMM and MMEW. Pre-training has been done on a new composition of five facial macro expressions datasets CK+, MUGFE, OuluCasia, SFEW, and RAF-DB. The proposed models are fine-tuned on the target ME dataset rigorously with Stratified 5-Fold and 10-Fold, Leave-One-Subject-Out, and Leave-One-Dataset-Out(LODO) cross-validations(CV). In all evaluations, our proposed algorithms show remarkable improvement in effectiveness which surpasses the state-of-the-art accuracies and results in higher generalization capacity.

Data-driven method has been widely used in Fluid Catalytic Cracking (FCC) process modeling. However, due to the complexity of chemical process both in time and spatial domain, how to reflect the time and spatial characteristics of FCC units and build corresponding model is important to construct a better model for the gasoline yield prediction. In this paper, a special neural network structure was developed to deal with the input variables with different time scales considering the collection characteristics of various variables, as well as the time continuity of large-scale process manufacturing units, LSTMs with different time scales are stacked to extract temporal and spatial features to help capture the relationship between influencing factors and product yield. The characteristics of FCC process are also fully reflected in data processing and building model. It is demonstrated from the conclusions that the new model developed in this paper performs better than the traditional LSTM networks, which will be of great help to the intelligent upgrading of the FCC process.

Facial Micro-Expression recognition in the field of emotional information processing has become an inexorable necessity for its exotic attributes. It is a non-verbal, spontaneous, and involuntary leakage of true emotion in disguise of most expressive intentional prototypical facial expressions. However, it persists only for a split-second duration and possesses fainted facial muscle movements that make the recognition task more difficult with naked eyes. Besides, there are a limited number of video samples and wide-span domain shifting among datasets. Considering these challenges, several video-based works have been done to improve the classification accuracy but still lack high accuracy. This works addresses these issues and presents an approach with a deep 3D Convolutional Residual Neural Network as a backbone followed by a Long-Short-Term-Memory auto-encoder with 2D convolutions model for automatic Spatio-temporal feature extractions, fine-tuning, and classifications from videos. Also, we have done transfer learning on three standard macro-expression datasets to reduce over-fitting. Our work has shown a significant accuracy gain with extensive experiments on composite video samples from five publicly available micro-expression benchmark datasets, CASME, CASMEII, CAS(ME)2, SMIC, and SAMM. This outweighs the state-of-the-art accuracy. It is the first attempt to work with five datasets and rational implication of LSTM auto-encoder for micro-expression recognition.

Accurate and real-time recognition of skeleton-based dynamic hand gestures has gained increasing attention in recent years with the development of depth sensors and the improved hand joints estimation algorithms. This task is challenging due to the spatial and temporal features that exacerbate the task complexity. Although previous works have applied different techniques, it remains challenging to efficiently and simultaneously encode the spatiotemporal features. To address this problem, this work presents a Deep Convolutional LSTM (DConvLSTM) model to learn more discriminative spatiotemporal features from skeleton data implicitly. The model employs multi-layer ConvLSTM to accurately capture the multiscale spatial and sequential information of the gesture and preserves a fast inference and lightweight size. Extensive experiments on three publicly available datasets show strong performance and demonstrate the superiority of our method by outperforming other methods. Furthermore, our method can achieve comparable recognition accuracy while maintaining small models and short inference time.

In affective computing, several deep learning-based strategies have been developed to classify facial micro-expression (ME), but the high recognition accuracy is yet to achieve due to some inherent challenges such as the low intensity of facial micro movement, region-specific changes, fraction second longevity, and inconsistency and a limited number of samples in publicly available spontaneous datasets. In this paper, we attempt to address these issues and propose a highly effective end-to-end deep model to recognize micro-expressions based on apex frames. Two-stage transfer learning through Image-Net and four macro expression datasets, and fine-tuning on four spontaneous micro-expression benchmark datasets, namely CASME, CASMEII, CAS(ME)2, and SAMM with four validation protocols have been implemented. Our experimental results surpass the effectiveness of the state- of-the-art methods and express the higher model generalization, which subsequently can expedite the applications such as lie catching, homeland securities, criminal detections, business deal negotiations, and clinical diagnosis through psychoanalysis.

The imbalanced data problem exists in many real-world datasets. Neural networks are one popular method for classifying imbalanced data. However, data imbalance often negatively affects neural networks. This problem is exacerbated when data are highly imbalanced. Existing neural network approaches for handling this problem heavily rely on rebalancing or reweighting known data. Essentially, these strategies focus on recovering the characteristics of balanced data. However, due to the serious lack of positive samples, the problem of insufficient empirical representation has not been thoroughly considered. Therefore, to solve the problem of highly imbalanced data, we explore the characteristics of the gradient norm in gradient descent optimization. We find that the key indicator of balanced data is that the gradient norms of positive and negative classes are approximately equal. Specifically, neural networks can classify known data that is highly imbalanced by considering the unit gradient direction of positive and negative classes. Furthermore, a local boundary expansion strategy is considered to alleviate the insufficient empirical representation problem of the positive class. Here, we propose a controllable gradient rotation strategy to realize local boundary expansion for positive samples. We validate the proposed approach on 34 highly imbalanced datasets and two synthetic datasets, and the proposed method exhibits impressive performance.

Hand gesture and action recognition have been extensively researched in the past two decades due to the emerging advanced acquisition and interaction technologies, which open the floodgates for a vast range of potential applications. Particularly, many spatial–temporal feature extractors have been proposed, such as RNNs-based models, temporal convolutional network (TCN), and 3D convolutional neural networks (3DCNN) for modeling long-term dependencies in sequential data. However, it remains challenging to obtain a high recognition rate because of the difficulty of effectively extracting spatial–temporal features and efficiently classifying them with noisy and complex skeleton sequences. Therefore, this paper proposes a deep ensemble framework called multi-model ensemble gesture recognition network (MMEGRN) for skeleton-based hand gesture recognition. Specifically, to establish effective feature extraction and accurate gesture recognition, we propose an architecture consisting of four sub-networks, three spatio-temporal features classifiers to leverage their various capabilities of extracting and classifying skeleton sequences. Through late feature fusion, the features resulted from the feature extractors of each sub-network are fused into a new fusion classifier. Each subnetwork is trained independently to perform the task of gesture recognition using only skeleton joints. The training is performed using the cyclic annealing learning rate to generate a series of models that are combined in an ensemble using the optimized weighted ensemble (OWE) method. The proposed framework combines deep learning and ensemble strengths to establish a new deep-learning network architecture for more accurate and efficient hand gesture recognition. Extensive experiments on three skeleton-based hand gesture recognition datasets have shown the effectiveness of the proposed framework and the superiority over other models in terms of recognition accuracy.

The industrial internet has become one of the “seven new infrastructures”, and its importance is self-evident. 5G networks can further promote the communication between people, people and things, things and things. The 5G-enabled industrial internet will make full use of the characteristics of 5G networks, including large bandwidth, low latency and wide coverage, to enhance the informatization of industrial systems. At the same time, the 5G networks have higher security technology level and stronger controllability. The integration of 5G networks and industrial internet will bring new security risks. Customized security protection capabilities such as network security isolation, terminal access authentication and end-to-end data security should be provided for different 5G+ industrial internet scenarios. In future, the thinking mode of the security protection of 5G-enabled industrial internet should turn to be active and intelligent.

In this paper, we collect an ocular B-ultrasound image dataset and propose a Collaborative Monitoring Deep Learning (CMDL) method to detect cataract. In the ocular B-ultrasound images, there are often strong echoes near the posterior capsule and lens, and the fine-grained ocular B-ultrasound images are often accompanied by the characteristics of weak penetrating power, low contrast, narrow imaging range, and high noise. Thus, in the proposed CMDL method, we introduce an object detection network based on YOLO-v3 to detect the focus areas we need, so as to reduce the interference of noise and improve the cataract detection accuracy of our method. Considering that the B-ultrasound image dataset we collected is small-scale, we also design three feature extraction modules to avoid over-fitting of the deep neural networks. Among them, there are a depth features extraction module based on DenseNet-161, a shape features extractor based on Fourier descriptor, and a texture features extraction module based on gray-level co-occurrence matrix. Moreover, we also introduce the collaborative learning module to improve the generalization of the proposed model. Specifically, we first fuse the depth, shape, and texture features of the eyeball and lens, respectively. Then, the fused features of the eyeball and lens are concatenated as the input of collaborative network. Finally, the introduced classification loss with the aid of collaborative loss, which distinguishes whether the eyeball and lens belong to the same category, improves the classification accuracy in cataract detection. Experimental results on our collected dataset demonstrate the effectiveness of the proposed CMDL method.

Recently, Chinese poetry generation gains many significant achievement with the development of deep learning. However, existing methods can not generate Chinese palindrome poetry. Besides, there is no public dataset of Chinese palindrome poetry. In this paper, we propose a novel Chinese palindrome poetry generation model, named Chinese Palindrome Poetry Generation Model (CPPGM), based on the universal seq2seq model and language model with specific beam search algorithms. In addition, the proposed model is the first to generate Chinese palindrome poetry automatically, and is applicable to other palindromes, such as palindrome couplets. Compared with several methods we propose, the experimental results demonstrate the superiority of CPPGM with machine evaluation as well as human judgment.

In recent years, the automatic generation of classical Chinese poetry has made great progress. Besides focusing on improving the quality of the generated poetry, there is a new topic about generating poetry from an image. However, the existing methods for this topic still have the problem of topic drift and semantic inconsistency, and the image-poem pairs dataset is hard to be built when training these models. In this paper, we extract and integrate the Concrete and Abstract information from images to address those issues. We proposed an infilling-based Chinese poetry generation model which can infill the Concrete keywords into each line of poems in an explicit way, and an abstract information embedding to integrate the Abstract information into generated poems. In addition, we use non-parallel data during training and construct separate image datasets and poem datasets to train the different components in our framework. Both automatic and human evaluation results show that our approach can generate poems which have better consistency with images without losing the quality.

As a written language used for thousands of years, Ancient Chinese has some special characteristics like complex semantics as polysemy and the one-to-many alignment with Modern Chinese. Thus it may be translated in a large number of fully different but equally correct ways. In the absence of multiple references, reference-dependent evaluations like Bilingual Evaluation Understudy (BLEU) cannot identify potentially correct translation results. The explore on automatic evaluation of Ancient-Modern Chinese Translation is completely lacking. In this paper, we proposed an automatic evaluation metric for Ancient-Modern Chinese Translation called DTE (Dual-based Translation Evaluation), which can be used to evaluate one-to-many alignment in the absence of multiple references. When using DTE to evaluate, we found that the proper nouns often could not be correctly translated. Hence, we designed a new word segmentation method to improve the translation of proper nouns without increasing the size of the model vocabulary. Experiments show that DTE outperforms several general evaluations in terms of similarity to the evaluation of human experts. Meanwhile, the new word segmentation method promotes the Ancient-Modern Chinese translation models perform better on proper nouns’ translation, and get higher scores on both BLEU and DTE.