Stereo matching, a pivotal task in 3D visual computing, aims to extract depth information for various multimedia applications. Despite its significance, deploying high-fidelity models on edge platforms remains challenging due to high memory bandwidth usage and redundant computational overhead. This paper introduces a computationally efficient stereo matching architecture, termed MCFNet, designed to minimize processing bottlenecks without compromising accuracy. To overcome feature extraction limitations, we propose a Multi-Frequency Contextual Interaction (MFCI) block that dynamically modulates local details with global context. Furthermore, a Context-Guided 3D Residual Aggregation (CGR-3D) module is introduced to optimize cost-volume processing by enlarging the receptive field via a lightweight context stream. Finally, a Feature-Level Multi-Sample Accumulation (FLMSA) module acts as a low-latency alternative for precise disparity refinement. Experimental results demonstrate that the proposed architecture achieves a highly competitive trade-off between accuracy and inference speed, securing a D1-all error rate of 2.73
Intelligent monitoring of animal behaviors aims to effectively observe and analyze animals’ activities by leveraging multiple technologies of information processing and analysis, which can significantly enhance the wildlife protection, ecosystem maintenance, and so on. There have been a considerable amount of research works in this field, but species diversity, complex environments, and behavioral variability also present substantial challenges. Therefore, this paper systematically reviews recent advancements in intelligent animal behavior monitoring techniques, so as to provide essential theoretical and technical guidance for related researchers. Specifically, we comprehensively analyze related works in two primary tasks of animal behavior recognition and pose estimation using traditional algorithms and deep learning techniques, and systematically explain the application of unmanned aerial vehicles (UAV) to intelligent detection utilizing various perspectives such as timeline diagrams, partitioned circular diagrams, and method flowcharts. We further summarize and categorize publicly available datasets related to intelligent animal monitoring, providing a comprehensive resource overview for related studies. Furthermore, we summarize the characteristics of core architectures from a new perspective combined the algorithms with visualization interfaces, outline the technical details and practical effects of models, and systematically compare the performance of different methods on several key indicators. Finally, we discuss remaining challenges and outline promising future research directions. This comprehensive review on animal behavior intelligent monitoring aims to systematically summarize the latest advanced technological frontiers of advanced artificial intelligence in animal behavior monitoring, also to identify the research challenges and explore future research directions in this field.
Existing video synopsis methods often struggle to preserve interactions between objects accurately and tend to generate many pseudo-collisions, which makes the summarized video difficult to interpret. To address these issues, we propose a surveillance video synopsis method that fuses object interaction awareness with spatio-temporal rotation. First, an object interaction function is defined to analyze inter-object relationships and group interactive tubes for joint processing. Then, collisions between objects are analyzed, and either dynamic temporal translation or spatio-temporal rotation is applied according to an object-scale threshold. An angle threshold is further introduced to divide spatio-temporal rotation into adaptive spatio-temporal rotation and critical spatio-temporal rotation, thereby reducing collisions while maintaining the synopsis rate. A timing function is also designed to preserve temporal coherence between objects. Finally, the object tubes are stitched and fused with the video background to generate the synopsis video. Experimental results on multiple datasets show that the proposed method achieves the highest average F-score of 0.6571 for interactive behavior preservation. It also attains a favorable average frame synopsis rate of 0.2277 and an overlap ratio of 0.0409, while maintaining faster processing speed and better subjective visual quality.
The specification of video-game behavior is commonly expressed through general-purpose scripting, coupling behavior definition to the components, callbacks, and APIs of a particular engine. This paper introduces GameRule, a multi-agent architecture for game behavior specification whose programming model is expressed through a domain-specific language (DSL) and realized by a reusable engine backend. Behavior is declared per actor as local when-do rules over a uniform property namespace covering the current actor, other actors, and the global game state. State access and transformation are expressed through a closed set of CRUD-based operations (Create, Read, Update, Delete), including higher-level actions whose engine-specific realization is delegated to the backend. Following the multi-agent system (MAS) paradigm, rules are declared locally to each actor but evaluated under a deterministic execution contract: a central scheduler traverses active actors in declaration order, actor evaluations are non-interleaved, and writes become immediately visible in execution order. The architecture is realized in Unity through a code generator that parses GameRule descriptors and produces C# scripts and configured game objects. Validation uses reimplementations of three games drawn from official Unity tutorials: John Lemon’s Haunted Jaunt, Survival Shooter, and Tanks. The three games use six conditions and fourteen actions without game-specific C# programming and show that the current model can represent and execute the gameplay mechanics and actor behaviors exercised by the tutorials in Unity. The evaluation also identifies recurring engine-specific responsibilities implemented once in the reusable backend, runtime cost and scaling limits, and behavior outside the model. Only the Unity backend has been implemented, so realization in other engines remains an architectural extension point rather than demonstrated portability.
Multimodal recommendation systems integrate multimodal information into user and item feature representations, and leverage multimodal information to alleviate the data sparsity issue. However, existing research faces two key challenges. First, most methods based on graph convolutional network (GCN) directly utilize multimodal features, which may introduce noisy information unrelated to user preferences. Second, existing diffusion-based methods primarily focus on interaction modeling or graph structure augmentation, which inevitably adds new noisy edges (e.g., introducing false user–item interactions). Although these methods have achieved remarkable success, the aforementioned noise can be amplified in the message propagation of GCN, affecting the final multimodal representation. To address these challenges, we propose a novel Diffusion-based Representation Augmentation for Multimodal Recommendation (DRARec) method. Specifically, to mitigate the negative impact of noisy information across different modalities, we introduce a behavior-aware injection mechanism. It first injects behavior modality information into all content modality embeddings, aligning them to a preference-related semantic space, and then employs an attention module to enhance feature discriminability. Additionally, to avoid the negative impact of noisy edges, we propose a diffusion-based modality representation augmentation strategy. It directly refines the semantic features of each modality to enhance the robustness of features. Extensive experiments on three public datasets demonstrate the superiority and effectiveness of DRARec.
Emotion perception is central to enabling proactive interaction in intelligent robots. However, in complex dynamic scenarios, random fluctuations in perception quality, semantic conflicts among multimodal information, and the absence of uncertainty modeling severely limit the robustness of existing systems. To address this challenge, this paper proposes a robust dual-modal emotion recognition framework (DW-DS ER) based on dynamic weighting Dempster-Shafer evidence theory. This framework mimics human cautious reasoning under conflicting cues by introducing an innovative adaptive trust allocation mechanism at the decision layer: it jointly models prior reliability and instantaneous conflict intensity across modalities, and using nonlinear mapping functions to adaptively weight multimodal evidence in regions of semantic conflict, the system dynamically selects the dominant information that contributes most optimally. This effectively suppresses systematic misjudgments caused by unimodal degradation. Furthermore, this paper constructs the Face-Pose Multimodal Evaluation (FEPE) dataset to systematically validate the proposed method’s performance across seven basic emotions. The experimental results show that DW-DS ER significantly outperforms the single-modality baseline, as well as classical decision-level and lightweight feature-level fusion methods, in terms of accuracy, F1 score, and uncertainty suppression. Moreover, the system achieves an end-to-end inference latency of only 47 ms, delivering exceptional edge computing efficiency while ensuring high robustness and interpretability. This provides an ideal engineering solution for trustworthy interactions by service robots in uncertain environments.
Recommendation systems are essential tools for delivering personalized content in diverse applications. However, in learning personalized preferences, most existing recommendation systems often fail to effectively capture the complex relationships between users and items, resulting in suboptimal recommendation outcomes. To address this challenge, we propose a novel neural network called the Co-occurrence Graph Neural Network (CoGNN), which utilizes two co-occurrence graphs to establish user and item relationships. In CoGNN, we designed two approaches to maximize learning from user-item interactions: co-occurrence graphs and Autoencoders for user behavior preferences. First, co-occurrence relationships are constructed using graphs, where noise is mitigated by a Normalized Point-wise Mutual Information (NPMI) matrix. Next, Autoencoders are employed to learn users’ behavior embeddings, extracting latent information from historical interactions between users and items. Finally, experimental results across multiple datasets in recommendation tasks consistently demonstrate that CoGNN outperforms various baseline models in terms of recommendation accuracy and algorithm convergence. Our code and datasets are publicly available at https://github.com/lyg2618/CoGNN .
Existing multimodal sentiment analysis (MSA) methods still face challenges such as insufficient fusion depth, limited cross-modal interactions, and unstable feature alignment, which may hinder robust representation learning in complex scenarios. To address these challenges, we propose a unified framework termed HiPDA (Hierarchical Perceiver-style Injection and Dual-Anchor Alignment), which organizes multimodal modeling in a progressive injection–fusion–alignment manner. The Hierarchical Perceiver-style Injection (HPI) introduces acoustic and visual information into intermediate layers of a language model, providing a controllable mechanism for modeling modality asynchrony while preserving textual semantic dominance. The Hierarchical Temporal Perceiver Fusion (HTPF) performs latent-space multimodal interaction and projects fused information back to textual representations, while the Dual-Anchor Consistency Alignment (DCA) constrains textual and fused representations to improve cross-modal consistency. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate the effectiveness of HiPDA. Compared with representative baselines, HiPDA improves non-zero ACC2 and ACC7 by 1.54 and 4.88 percentage points on CMU-MOSI, respectively. On CMU-MOSEI, it improves non-zero ACC2 and ACC7 by 0.84 and 0.13 percentage points, respectively. On CH-SIMS, HiPDA achieves strong ACC2, ACC5, MAE, and Corr results, while its ACC3 behavior is further analyzed through class-level evaluation. These results indicate that HiPDA provides a structured and effective framework for multimodal sentiment modeling.
Character art represents a complex intersection of visual aesthetics and s evolution, posing a significant challenge for Multimodal Large Language Models (mLLMs). Effective analysis requires more than surface-level description; it demands “deglyphion”—a structured descriptive analysis that grounds visual forms in historical script evolution while distinguishing artistic intent (e.g., pictographic vs. abstract design). General-purpose mLLMs often produce fluent but “hallucinated” narratives because they lack the domain-specific evidence necessary to link modern artistic glyphs to their ancient precursors. To bridge this gap, we propose an evidence-grounded framework that conditions mLLM generation using a Structured Analytical Profile (SAP). This framework employs a two-pronged evidence extraction strategy: (i) a specialized vision-language classifier (LLaVAsty) that categorizes artistic style and (ii) a Siamese metric-learning model that retrieves structurally similar exemplars from a curated historical glyph knowledge base to provide morphological context. These signals are synthesized into the SAP, which serves as a machine-readable interface to guide foundation mLLMs via in-context learning. We evaluate our framework on a newly constructed dataset comprising 14,740 Tianshu images and a historical repository of 12,600 labeled glyphs. Experimental results across multiple state-of-the-art mLLMs demonstrate that SAP conditioning significantly improves visual faithfulness and historical grounding. Controlled ablations confirm that the integration of historical retrieval and stylistic priors reduces semantic overreach and enhances the scholarly utility of the generated deglyphions. Our work provides both a novel methodology for computational art history and a robust resource for the analysis of non-standard scripts.
Industrial anomaly detection for oil and gas pipelines faces severe challenges due to unstructured defects and highly dispersed intra-class semantics, especially in zero and few-shot scenarios. Existing vision–language methods rely on fixed or learnable prompts and single-image training frameworks, which limit the modeling of diverse anomaly features and stable representation learning. To address these issues, we propose ElainaCLIP, a prompt learning framework based on CLIP. Specifically, we design ElainaPrompt to incorporate vision-guided information into text prompt representations and introduce ElainaLoss to guide prompt learning through low-level semantic constraints, thereby enhancing the modeling of unstructured anomaly semantics. Meanwhile, a Multi-image Joint Optimization Strategy is introduced to exploit semantic synergy among samples and improve representation consistency under diverse intra-class semantics. We also construct PipeAD, a dataset with pixel-level labels for pipeline anomaly detection to fill the data gap. Extensive experiments show that our method improves image-level and pixel-level AUROC by 5.6 https://github.com/ElainaV/ElainaCLIP .
Blind super-resolution (SR) aims to recover high-resolution (HR) images from low-resolution (LR) observations without prior knowledge of the degradation process. Recent deep unfolding methods offer a promising hybrid paradigm by integrating iterative optimization with neural networks, yet they still face limitations in efficiently leveraging explicit degradation priors and accurately estimating blur kernels. To address these issues, we propose a novel deep unfolding network that combines Fourier-domain closed-form solvers with a Cascaded Residual Network (FDCRNet). Specifically, we formulate blind SR as a joint optimization problem over the HR image and blur kernel, decomposed via half-quadratic splitting into alternating data fidelity and prior regularization subproblems. For the data fidelity subproblem, we derive a closed-form solution in the Fourier domain, enabling efficient kernel estimation while maintaining high reconstruction fidelity. For the prior subproblem, we introduce a Cascaded Residual Fusion Network (CRFNet) to enhance prior modeling by effectively integrating multi-stage and multi-level features. Extensive experiments on standard benchmarks demonstrate that the proposed model outperforms state-of-the-art methods in both quantitative metrics and visual quality, offering a promising solution for real-world blind SR.
Infrared–visible object detection aims to fuse the complementary information from infrared and visible images to achieve robust perception in complex scenes and enable all-weather object detection. However, existing methods still encounter significant challenges in modality alignment and semantic interaction. In complex scenes, environmental interference often leads to shifts in the feature semantic space, weakening the semantic consistency across modalities. Additionally, feature fusion struggles to fully capture cross-modal semantic dependencies and structural complementarity between infrared and visible representations. To address these issues, we propose a Layer-Enhanced Collaborative Fusion Detection Transformer (LECF-DETR). Specifically, the Layer-wise Feature Enhancement Module (LFEM) is designed to adopt differentiated enhancement strategies for features at different scales, improving the semantic consistency and discriminability of multimodal features and alleviating semantic shifts caused by environmental interference in complex scenes. Meanwhile, the Multimodal Collaborative Fusion Module (MCFM) is introduced to effectively explore the semantic dependencies and structural complementarity between modalities, generating more consistent fused representations. Finally, extensive experiments on the LLVIP, FLIR, and M3FD datasets demonstrate that LECF-DETR achieves competitive and robust detection performance compared with existing methods across different multimodal benchmarks.
Camera relocalization estimates the 6-DoF pose of a query image within a prebuilt scene and remains challenging under viewpoint changes, weak textures, and repetitive structures. Existing hybrid methods combine sparse pose recovery and dense refinement but rely on disjoint scene representations. This paper presents LoFG, a localization-oriented Feature Gaussian representation that unifies both stages within a single Gaussian scene. LoFG augments Gaussian primitives with feature embeddings and induces compact landmarks for robust initial pose recovery. Using the same representation, it renders dense feature fields and depth maps for high-precision refinement. Experiments on 7-Scenes and Cambridge Landmarks show that LoFG reduces median translation error to 0.87 cm and rotation error to 0.26 degrees indoors, outperforming baseline methods. The unified design improves the robustness of sparse initialization and the accuracy of dense refinement, demonstrating potential for localization applications in AR, robotics, and visual navigation systems. The source code and documentation associated with this manuscript are available at https://github.com/auXiao2022/LoFG .
The main challenges in Camouflaged Object Detection (COD) include the high similarity between the object and the background in complex scenes, as well as the limitations of existing methods in feature processing. There are two major problems in current COD methods. One is that the single feature processing method leads to insufficient feature information capture, which affects the accuracy of target localization. The second is that weak and ambiguous object boundaries make it difficult to achieve accurate object segmentation. In addition, the improvement of performance is often accompanied by the increase of the number of parameters and the rise of computational complexity. To solve these problems, this paper proposes an Adaptive Feature Interaction and Edge Refinement Network (AFNet), a lightweight hierarchical feature refinement framework that efficiently aggregates multi-level features to improve object localization and enhance boundary-related representations while balancing performance and computational cost. Specifically, the Multi-scale Fusion Enhancement Module (MFEM) is introduced to progressively refine cross-level features and enhance feature discrimination under high foreground-background similarity. At the same time, the Edge Fusion Decoding Module (EFDM) is designed to perform lightweight feature-level boundary refinement by exploiting boundary cues from decoder features, preserving feature information and further improving the camouflage prediction results. Experimental results show that AFNet achieves competitive performance compared with 21 state-of-the-art methods on three challenging datasets. It is worth mentioning that AFNet maintains a lightweight structure with 12.3M parameters and 9.7G FLOPs, while achieving an inference speed of 177.1 FPS.
Contrastive learning based dual-encoder architectures demonstrate substantial progress in cross-modal audio-text retrieval. In practical retrieval settings, the importance of temporal segments depends on query semantics. For the same audio clip, different textual descriptions may refer to different acoustic events or temporal spans. An ideal matching mechanism should therefore capture a temporal salience distribution conditioned on text semantics. However, in standard dual-encoder retrieval, fine-grained temporal salience induced by text during training cannot be directly incorporated into the fixed audio representation used for retrieval. To address this issue, this work proposes Semantic-Conditioned Temporal Structure Distillation, SCTSD. Rather than relying on an additional pretrained teacher model, SCTSD constructs an online text-conditioned teacher branch within the retrieval architecture. During training, the teacher branch uses paired text descriptions to generate temporal salience distributions over multi-scale audio tokens and transfers this structural supervision to a student branch whose salience prediction and audio pooling rely only on audio input. During inference, the teacher branch and its text-conditioning mechanism are removed. Only the audio-only student is retained to produce a fixed embedding for each audio clip that is independent of the test query. SCTSD therefore preserves the standard dual-encoder retrieval pipeline while learning temporal structural information induced by text conditions during training. Experimental results on AudioCaps show that SCTSD improves R@1 from 38.9
Deep multi-view clustering aims to integrate complementary information from multiple heterogeneous views to improve clustering performance. However, existing methods are often constrained by two key bottlenecks. On the one hand, although mainstream contrastive learning methods can capture cross-view consistency, they often neglect the intrinsic local geometric structure of data, resulting in feature representations with limited inter-cluster discriminability. On the other hand, methods that rely on pseudo-labels for semantic guidance, while promising, are highly susceptible to noisy pseudo-labels, causing the model to overfit incorrect pseudo-labels generated in the early training stage and severely degrading clustering quality. To address these issues, we propose a structure-aware and noise-robust multi-view clustering method. Specifically, we introduce graph-structure-aware feature fusion, which jointly optimizes instance features and local manifold structures through KNN graphs and graph convolution, thereby enhancing inter-cluster separability. Meanwhile, we propose a K-Means-induced adaptive curriculum learning mechanism that selects reliable samples using multi-view averaged entropy confidence and dynamic quantile thresholds, suppressing noise interference and enabling robust batch refinement. Experimental results on five benchmarks demonstrate that the proposed method achieves the best results on Scene15, CUB, and Reuters and remains competitive on NUS-WIDE and XMediaNet compared with 13 representative clustering methods, including ROLL, MVCAN, SCM-RE, and LargeMvC-Net.
Multi-modal image fusion seeks to integrate complementary information from heterogeneous sensors, yet existing Transformer-based approaches typically delegate structural preservation, cross-modal feature selection, and multi-scale interaction to a single self-attention pathway. This homogeneous design forces the network to compromise between competing objectives, resulting in blurred edges and redundant modal information. We propose EdgeAttnSwin, a framework that reformulates fusion as a three-stage progressive optimization process with explicit causal dependencies. An Edge-Guided Mechanism injects multi-directional gradient priors as hard geometric constraints. A serial Dual-Attention Module then performs channel-first, spatial-second recalibration to suppress cross-modal redundancy while preserving salient targets. Finally, a Hierarchical Multi-scale Aggregation enforces geometric consistency across resolutions via bidirectional feature pyramid propagation. These three stages form a collaborative pipeline where each stage serves as an indispensable prior for the next. Extensive experiments on infrared-visible and medical image fusion benchmarks demonstrate that our method outperforms state-of-the-art approaches, achieving 6.6 https://github.com/wushangsuke-hub/EdgeAttnSwin .
User authentication is more and more crucial in everyday life, with researchers developing sophisticated, yet easy-to-use, approaches to enforce secure authentication while providing a smooth user experience. This is particularly true in mobile authentication, in which the spread of AI-capable smartphones is opening up several biometric-based authentication methods, such as face and lips recognition. A more recently emerging setup consists of the matching of a spoken passphrase with lips movements. Despite being promising and well-suited for mobility authentication, the matching operation is not trivial, and the research around this topic is limited, as no publicly available dataset is available for experimental purposes. In this paper we thus introduce BioVid, a novel privacy-aware multimodal biometric dataset for user recognition from smartphone-recorded videos in real-world conditions. BioVid consists of 650 recordings from 43 participants, where each clip captures a user pronouncing predefined passphrases while simultaneously acquiring audio and lips movements. The dataset is designed to support dual-factor biometric authentication, combining speaker-unique identity (UID) and passphrase verification, and explicitly models realistic variability in device type, recording distance, background, and ambient noise. Together with the data, we provide a detailed statistical analysis of the linguistic, visual, and acoustic properties of the data, highlighting its diversity and suitability for robust biometric system evaluation. Moreover, to test the quality of the collected dataset, we organised an international challenge on multimodal user authentication leveraging BioVid. In this paper we thus also describe the challenge protocol, baseline systems, and evaluation metrics, while also summarising the best-performing submitted approaches as measured by Equal Error Rate (EER). BioVid and the accompanying challenge aim to establish a reproducible benchmark for multimodal biometric authentication on consumer-grade smartphones and to foster further research on privacy-aware, real-world biometric systems.
Extended Reality (XR) enhances cultural experiences by immersing users in virtual environments where virtual assets and 3D reconstructions provide a stronger connection to real-world scenarios. In this work, we present a platform for collaborative immersive cultural experiences, focused on real-time interaction between participants. The platform provides three main features designed to enable meaningful communication inside of virtual scenarios: volumetric representation of a presenter captured in real time, virtual scene control through gesture recognition for the presenter, and bidirectional audio as well as 3D avatar representation for every user. The platform was integrated into a real-world 5G network to test its capabilities as well as validating the ability of the infrastructure to handle demanding uplink bitrate and computation-intensive XR applications. Both network- and application-level experiments were conducted to measure key performance metrics, including latency, packet loss, and throughput. The results demonstrate the feasibility of the proposed platform and its ability to support real-time communication in collaborative XR scenarios.
Semantic segmentation is a fundamental task in diverse fields such as remote sensing, medical diagnosis, and infrastructure inspection. However, many advanced segmentation models are highly specialized for particular data characteristics, making it difficult to maintain a favorable balance between local boundary detail, global context, and computational cost across different application domains. To address this challenge, this study proposes DFC-SA-UNet, a U-Net-based semantic segmentation network built around a Dynamic Fusion Convolutional Self-Attention Block (DFC-SA Block). This block synergizes the local feature extraction capability of convolution with the global context modeling of a Lightweight Self-Attention module. A dynamic gating mechanism adaptively fuses these heterogeneous features, enabling the model to adjust its emphasis between local and contextual cues according to the input feature distribution. We evaluate the framework on three datasets from different domains: satellite image building segmentation, medical colon polyp segmentation, and civil infrastructure wall crack detection. The results show that DFC-SA-UNet achieves consistently competitive performance compared with U-Net, ViT-Seg, TransUNet, and SegFormer-B5 under a repeated per-dataset train/validation/test evaluation protocol, while maintaining a parameter count comparable to U-Net and lower than those of the Transformer-based baselines. These experiments support the effectiveness of the DFC-SA Block as a lightweight adaptive fusion module for multi-domain semantic segmentation, while broader cross-domain generalization evaluation is left for future work.
