Image fusion generates more information-rich integrated images by integrating multi-source image data, and boasts extensive applications in fields such as environmental monitoring and battlefield reconnaissance. Although existing technologies can achieve favorable multi-modal image fusion in non-interfering scenarios and aid in the presentation of scene information, they exhibit significant limitations when confronted with the degradation of low-quality source images induced by extreme scenarios, including low light, haze, rain and snow, motion blur, and strong noise interference. To tackle these challenges, this paper proposes a novel anti-degradation multi-modal image fusion framework guided by supplementary text instructions. Firstly, the Low-rank mechanism is employed to conduct targeted fine-tuning on the CLIP vision-language large model, enabling accurate capture of scene degradation features such as low light, noise, and motion blur, and facilitating the generation of semantic descriptions. Secondly, a heterogeneous mixture-of-experts feature extraction backbone network is designed, which leverages scene degradation semantic descriptions to dynamically regulate heterogeneous expert modules. Meanwhile, a mutual information loss function is introduced to optimize the sparsity of experts during multi-task processing, thereby avoiding redundant computations. Finally, each module undergoes training-testing collaborative optimization to enhance the controllability of the fusion process and improve the generalization capability of the model. Experiments on 4 mainstream benchmark datasets (LLVIP, MSRS, M3FD, ROAD) demonstrate that the proposed text-semantic-guided sparse mixture-of-experts mechanism exhibits significant advantages over state-of-the-art methods in terms of image fusion performance and degradation correction. This research outcome provides new insights for multi-modal image fusion in complex degraded environments and is expected to promote technological advancement in fields such as remote sensing images and medical images.
Owing to the inherent limitation of coarse-grained supervision, contrastive vision-language models such as CLIP typically exhibit suboptimal performance in fine-grained attribute understanding tasks. We propose SubCLIP, a lightweight framework designed to enhance the capability of the CLIP in capturing subtle visual-semantic distinctions. SubCLIP enhances fine-grained attribute understanding by decomposing the positive and negative captions of an image into subtexts containing attribute words and corresponding nouns and introducing a multi-granularity contrastive learning objective. Specifically, we introduce Hierarchical Subtext Augmentation, which leverages a large language model to generate fine-grained, subtexts from existing annotations, thereby providing denser and more informative training signals. To fully exploit these subtexts, we propose Multi-Granularity Contrastive Learning, a unified strategy that jointly optimizes global image-text alignment, intra-sample attribute-level contrast, and subtext-level compositional alignment. This hierarchical design enables SubCLIP to learn both high-level semantic associations and subtle attribute differentiations. On the FG-OVD benchmark, SubCLIP significantly improves fine-grained attribute retrieval performance. When using ROI features for retrieval, it achieves Top-1 accuracy rates of 62.4 % on the hard samples, surpassing previous state-of-the-art methods. Our results demonstrate SubCLIP's effectiveness in enhancing fine-grained vision-language understanding, delivering robust performance while maintaining computational efficiency and domain adaptability. Our code is available at https://github.com/mop134679852/SubCLIP.
Falls pose a serious challenge to the health and safety of the elderly population. In crowded public places, such incidents may also trigger secondary disasters such as stampede, so rapid identification of fall behavior is crucial for maintaining public safety. Traditional fall detection algorithms have significant deficiencies in terms of insufficient feature extraction, single detection method, and poor real-time performance, which make it difficult to cope with complex fall scenarios and real-time detection requirements. In order to overcome the shortcomings of traditional fall detection algorithms, this study improves the PPYOLOE algorithm. We optimize feature extraction using Depthwise Separable Convolution (DSConv) and introduce High-Level Screening-feature Fusion Pyramid Networks (HS-FPN) and Efficient Multi-Scale Attention (EMA) to enhance important features. The improved algorithm significantly improves the detection performance and computational efficiency. Compared with the traditional PPYOLOE algorithm, the fall detection algorithm designed in this paper shows significant advantages in detection accuracy and real-time performance. In the experiment, Our improved algorithm has achieved a 3.5% increase in accuracy on the mAP@0.5 metric, it reaches 94.4% and achieves 24fps on GPU, which realizes the high precision detection of fall events and meets the deployment requirements in low computation scenarios. The real-time and high precision of the algorithm makes it suitable for various monitoring scenarios, which improves the social security level and emergency response capability.
Accurate identification of crop diseases is crucial for enhancing agricultural yields. In the agricultural field, crop disease images vary in resolution and aspect ratio due to different imaging equipment, distances, and environmental conditions, which can affect the accuracy and consistency of disease detection models. We propose a new model, MMCA-ConvNeXt, based on ConvNeXt-T. This model incorporates our novel Multi-scale Multidimensional Collaborative Attention (MMCA) mechanism, which enhances the receptive field of the original model while capturing features at multiple resolutions. This enables better generalization across different crop disease datasets, improving performance and robustness. Additionally, we replaced layer normalization in the downsampling layer and post-global average pooling with Batch Channel Normalization, which allows better exploitation of inter-channel information and enhances gradient flow within the model. Experiments using publicly available datasets for rice, maize, and the 2018 AI Challenger demonstrate the superior performance of our ConvNeXt-based model, achieving accuracies of 99.53% for rice, 99.93% for maize, and 87.57% for the 2018 AI Challenger dataset. It outperforms VGG - 19, ResNet - 101, EfficientNet, and several transformer models with fewer parameters. MMCA shows the best results among different attention mechanisms for plant leaf disease detection, and ablation studies confirm the model’s performance enhancements.
In recent years, the research has been ongoing about crowd simulation. In many situations, crowd are usually organized into several groups and proceed to their respective target location, to accomplish something or perform a task together for a duration, and this process may involve communication and cooperation within the group. We define such crowd as a task-driven group cooperative crowd, and focus on the crowd movement and group behavior during this process. The movement process of the crowd involves multiple stages and exhibits different behaviors at different stage. Within this framework, we consider a problem scenario where crowd within the group orderly gathering in front of objects for a duration to perform a task. We propose a data-driven framework based on Multi-Agent Reinforcement Learning(MARL), including task parameter policy learning and curriculum training strategy. We use a set of task parameters to describe the current task information and integrate them into the iterative optimization process of the policy. Based on this, we design agent observation and a weighted reward function, to make the agent recognize the current task stage and learn the corresponding behaviors. Utilizing curriculum training strategy, the training process is divided into multi steps, gradually increasing the training complexity. We conduct a series of experiments to verify the effectiveness and generalization of the framework and design a museum exhibition scenario to verify the practicality. We provide a demo to supplement the more details, https://github.com/ICSRC/2024ICVRV.
