This work presents a configurable framework for generating synthetic optical flow datasets based on controlled motion scenarios. Built upon an extended Kubric environment, the framework allows users to specify object motions, camera trajectories, and optional environmental effects such as rain, fisheye distortion, and occlusions. Using this framework, we generate a dataset that separates object and camera motion into well-defined categories, and we additionally provide a higher-resolution version of the dataset. The motion scenario organization enables evaluation under clearly separated object and camera motion conditions. Several state-of-the-art optical flow models are evaluated on these scenarios to examine their behavior under different motion configurations. We also perform leave-one-scenario-out fine-tuning to study how targeted exposure to specific motion categories affects performance. Additional experiments assess model performance on fisheye-distorted images and on standard benchmarks such as MPI-Sintel and KITTI. The results highlight which scenarios are challenging for different models and show that fine-tuning on selected scenarios can influence performance on both synthetic and real datasets. The dataset generation framework and implementation scripts are publicly available at https://github.com/niloofarhp/Motion-Scenario-based-Dataset-FrameWork.git. Overall, the framework and dataset are intended to support systematic, controlled, and reproducible experimentation in optical flow research.
The widespread use of digital displays and mobile cameras has increased the risk of information leakage through screen capture and photography. Partial screen capture, in which only a cropped, zoomed, or reframed region of the displayed content is observed, introduces geometric distortions, content loss, and camera-display artifacts. These degradations disrupt the spatial correspondence between watermark embedding and extraction, resulting in synchronization loss and unreliable watermark recovery. Existing localization-based watermarking methods often use the information-bearing watermark itself as the localization cue, making it difficult to independently optimize watermark fidelity and localization robustness. To address this limitation, we propose RW-PSCDLD, a decoupled deep watermarking framework specifically designed for partial screen-capture scenarios. RW-PSCDLD separates information watermark embedding from dedicated localization watermark embedding, allowing the two objectives to be optimized independently. Within this framework, we introduce the Homomorphic Localization Watermark based on Keypoint Detection (HLWKD). HLWKD employs a lightweight generator to embed Gaussian-template-constrained localization patterns at the corners of the information watermark region. A keypoint detector combining multi-scale feature extraction, deformable attention, and high-resolution prediction is designed to accurately localize these patterns under geometric and photometric distortions. Experiments under simulated and real-world screen-capture conditions demonstrate the effectiveness of the proposed method. Compared with a strong coupled baseline, RW-PSCDLD improves the PSNR by 1.44 dB within the watermark region and by 1.47 dB globally. Under combined attacks, it improves localization IoU and bit accuracy rate by 1.5 and 5.7 percentage points, respectively. The proposed method also achieves a favorable balance among visual quality, localization accuracy, watermark robustness, and computational efficiency.
Single image super-resolution (SISR) aims to reconstruct high-resolution images with faithful structures and fine details from low-resolution inputs. Existing lightweight SISR methods often suffer from a fundamental trade-off between local detail reconstruction and efficient global dependency modeling. While window-based or pixel-wise attention mechanisms can enhance contextual interaction, their computational cost grows rapidly with spatial resolution, making efficient global reasoning challenging in lightweight architectures. In this work, we propose SPENet, a lightweight pattern-oriented super-resolution network that performs structured pattern embedding and efficient global dependency modeling. The core idea is to model repetitive and self-similar structures as reusable pattern representations, enabling lightweight long-range interaction in the channel domain rather than dense pixel-wise matching. Specifically, we design a Parallel-Hierarchical Pattern Embedding (PHPE) module to capture complementary fine-grained and abstract structural patterns through a dual-branch architecture. Based on these structured embeddings, we further introduce a Pattern Selective Global Attention (PSGA) module, which performs lightweight channel-wise indirect global interaction guided by a pattern selector. The selector enhances stable and informative structural responses while suppressing noisy or redundant activations, thereby alleviating the low-pass tendency commonly observed in conventional attention mechanisms. Extensive experiments on standard SISR benchmarks demonstrate that SPENet achieves state-of-the-art performance among lightweight models while maintaining favorable computational efficiency. Additional analyses on pattern-rich datasets, attribution responses, and frequency-domain behavior further indicate that the proposed framework is particularly effective for reconstructing repetitive and long-range correlated structures.
The U-shaped models have been extensively applied in medical image segmentation, with numerous researchers striving to boost model performance by aggregating features in the skip connections part. However, existing studies have not thoroughly investigated the intricacies of feature aggregation in skip connection, which leaves these aspects unresolved. Therefore, the feature aggregation adopted in skip connection has been revisited in this study. Specifically, two problems are explored. One is which encoder features are suitable for aggregation. The other is how to effectively aggregate the features. Accordingly, the proposed Multilayer Feature Aggregator (MLFA) and Decoder-Guided Interaction Adjustment (DGIA) module are the solutions of the mentioned problems. In addition, a novel network called MDI-Net is proposed based on the integration among MLFA, DGIA, and the pretrained InceptionNeXt block. Two categories of experiments are set up for testing. Firstly, both MLFA and DGIA are inserted into four popular U-shaped models to test the usefulness of the proposed modules. Secondly, the proposed MDI-Net is validated on four commonly used datasets, which are ISIC2018, BUSI, Kvasir-SEG, and CVC-ClinicDB. Experimental results demonstrate that the integration of MLFA and DGIA can improve the model performance and the proposed MDI-Net can outperform other popular models in medical image segmentation task. Code is available at https://github.com/shen123shen/MDI-Net-main.
Multimodal action recognition has attracted significant attention since the introduction of affordable multimodal sensors. While existing studies have demonstrated that integrating RGB and skeleton modalities can enhance recognition performance, these approaches typically depend on multi-stream architectures with a large number of parameters. Consequently, exploring efficient fusion strategies for RGB and skeleton data remains an important research direction. In this paper, we propose a novel Adaptive Feature Recalibration Network (AFRNet) that effectively integrates RGB and skeleton data at an early stage within a single-stream framework composed of two input branches. Compared with existing multimodal methods, our approach achieves competitive performance with substantial advantages in terms of model size and inference speed. Specifically, two independent encoders are first utilized to extract appearance and motion feature representations from RGB frames and skeleton sequences, respectively. These two types of features are then fused in the proposed adaptive feature recalibration module, where motion features are recalibrated by incorporating appearance information. The recalibrated features are subsequently processed by the main stream network to generate more discriminative representations. Additionally, we introduce a parameter-free adaptive key frame scan module to select the most informative frames from RGB videos. Experimental results on two large-scale benchmarks (NTU RGB+D 60 and 120) show that our AFRNet achieves state-of-the-art (SOTA) performance. For example, AFRNet obtains 94.9% accuracy on the cross-subject protocol of the NTU RGB+D 60 dataset, while being 2.4× smaller and 1.4× faster than SAM-Net, a leading SOTA method.
In long-term visual tracking, targets may temporarily disappear from the camera’s view, requiring robust re-detection to resume tracking. Existing re-detection methods often suffer from limited accuracy and struggle under challenging conditions such as scale change, cluttered backgrounds, occlusions, or long-term disappearance. To address these issues, we propose a global re-detection framework based on hierarchical feature fusion and target-guided perception. The framework consists of three complementary modules. The Hierarchical Feature Fusion Module integrates multi-level features through top-down and bottom-up fusion, enhancing multi-scale representations to better handle targets with significant scale variations. The Semantic Guided Cascade Perception Module strengthens the discriminative capacity of deep features and improves target recognition and localization under cluttered backgrounds and occlusions. The Target Guided Perception Module employs template-guided directional attention and global semantic modeling to enhance global re-detection and maintain robustness after long-term disappearance. The proposed re-detection module is integrated with six representative short-term trackers to construct their long-term variants. Extensive experiments on UAV20L, LaSOT, VOT2018-LT, and VOT2020-LT demonstrate significant improvements in success rate and precision. For example, the enhanced TransT achieves a 6.5% increase in success rate and a 4.8% improvement in precision on UAV20L, confirming the effectiveness and generality of the proposed approach.
Video instance segmentation aims to identify and segment target instances in video sequences while maintaining identity association across frames. However, due to inadequate temporal information modeling and inefficient memory management, existing methods cannot effectively handle complex scenarios such as occlusion, drastic motion, and large appearance changes. We propose an efficient online method using a trajectory-guided memory network (TGMN), which consists of a trajectory-appearance joint modeling (TAM) module and a temporal-aware memory update and management (TMUM) module to enhance the accuracy of instance segmentation and cross-frame association in complex scenarios. Specifically, the TAM module employs a temporal state modeling branch to capture complex motion patterns and a spatial appearance modeling branch to aggregate appearance information, effectively learning spatio-temporal feature representations of video instances. Additionally, we introduce a learnable threshold-based gating layer to suppress noisy trajectory during query propagation and utilize masked self-attention to efficiently fuse temporal and spatial features for robust spatio-temporal feature modeling. The TMUM module applies channel-spatial collaborative filtering to reduce noisy memory features across multiple dimensions. With a fixed-size memory bank, it updates memory based on appearance similarity and uses temporal positional encoding to enhance temporal correlation. Extensive experiments on three benchmarks demonstrate that TGMN achieves competitive performance compared to state-of-the-art methods. For example, TGMN achieves an AP score of 37.2 and 117.4 G FLOPs on OVIS.
While recent advances in reinforcement learning have significantly enhanced reasoning capabilities in large language models (LLMs), these techniques remain underexplored in multimodal large language models (MLLMs) for detailed video captioning. This paper presents a systematic investigation of GRPO-based RL post-training for MLLMs, with the goal of enhancing MLLMs’ capability of describing actions in videos. Specifically, VideoCap-R1 adopts a two-stage generation pipeline: it first produces a structured intermediate representation that enumerates the major video subjects, their visual attributes, and their temporally ordered actions, and then composes a detailed caption based on this representation. We optimize this behavior with GRPO using two complementary rewards: an LLM-free think scorer for the intermediate representation and an LLM-assisted event-coverage scorer for the final caption. The RL training framework effectively establishes the connection between structured reasoning and comprehensive description generation, enabling the model to produce captions with more accurate actions. Our experiments demonstrate that VideoCap-R1 achieves substantial improvements over the Qwen2-VL-7B baseline using limited samples (1.5k) across multiple detailed video captioning benchmarks (DREAM-1 K: +4.4 event F1, VDC: +5.0 Acc, CAREBENCH: +3.1 action F1, +6.9 object F1) while consistently outperforming the SFT-trained counterparts, confirming the effectiveness of GRPO-based training in enhancing MLLMs’ detailed captioning capabilities.
Recently, unsupervised learning has emerged as a significant direction in image dehazing research. However, existing methods still suffer from inherent limitations. In dual-branch cyclic frameworks, the generator processes inputs from different distributions simultaneously, leading to learning confusion. Moreover, physical disentanglement frameworks impose insufficient constraints on the haze components, thereby limiting their ability to accurately reconstruct scene details. Meanwhile, traditional channel attention mechanisms rely solely on global average pooling to extract channel representations, resulting in substantial information loss and consequently leading to suboptimal allocation of channel weights related to dehazing. To address the above issues, we propose a Self-Guided Region-aware Causal Collaborative Contrastive Network with Positional-Embedded Channel Attention (SGRPC-Net) for image dehazing. Specifically, we design a Self-Guided Region-aware Causal Collaborative Contrastive Framework (SRCF) that establishes region-aware collaborative contrastive constraints and causal counterfactual contrastive constraints. This framework not only alleviates the learning confusion inherent in dual-branch cyclic architectures but also enables adaptive regulation of dehazing intensity across different regions. Furthermore, it enhances the constraint capability on haze components at the feature level, thereby improving the network’s reconstruction performance. In addition, we design a Positional Embedded Channel Attention (PCA) module that integrates the captured horizontal and vertical positional information into channel representations, compensating for the loss of positional information caused by global pooling and enabling more precise channel weight allocation. Experimental results on challenging benchmark datasets demonstrate that our SGRPC-Net outperforms state-of-the-art unsupervised image dehazing methods. The source code is available at https://github.com/Lose-Code/SGRPC-Net.
Camera localization is the process of automatically determining the position and orientation of a camera with respect to its 3D environment based on the images it captures. Camera localization methods, including classical structure-based techniques and modern deep neural networks, often encounter limitations in visually complex environments when relying only on RGB images. Some of these limitations can be overcome by processing RGB-D images, provided that the available color and depth cues are properly integrated. Building upon CoAtNet, a successful hybrid Transformer neural model, this paper introduces CoAtXNet: a dual-stream architecture for jointly processing RGB color and depth channels. CoAtXNet interrelates two CoAtNet networks working in tandem through a straightforward cross-attention mechanism that intertwines the final Transformer stages. This integration of independent data streams yields enhanced feature representations. Experiments on the well-known 7-Scenes and 12-Scenes RGB-D camera localization datasets show that CoAtXNet achieves highly competitive performance, improving average translation and orientation accuracy over several recent localization methods. In addition to quantitative comparisons, we analyze the behavior of CoAtXNet under progressively degraded RGB appearance. The results show that the proposed model is less sensitive to appearance degradation than simpler RGB-D fusion baselines, suggesting that the cross-attention mechanism allows the model to place greater emphasis on geometric structures when RGB reliability decreases. An implementation of the proposed method is publicly available.
The proliferation of pre-trained models has significantly influenced the landscape of deep learning, providing a solid foundation for solving various downstream tasks. Recently, Knowledge amalgamation (KA) has emerged as a powerful framework for reusing pre-trained models by training a multi-talented student model. However, a key challenge, in this context, arises from that many pre-trained models are released without accompanying training datasets. In this paper, we address this challenge by introducing a novel paradigm called Out-of-Domain Knowledge Amalgamation (OOD-KA), which leverages solely out-of-domain data in the wild to learn the student model from pre-trained models. At the core of our proposed OOD-KA approach lies the Patch Amalgamation method, which enables the generation of task-specific data and features by capturing and reassembling local patterns in the OOD data. These amalgamated features are then utilized to effectively merge the knowledge from pre-trained models into the student model. We validate our method on image classification and semantic segmentation tasks across various benchmark datasets, and the experimental results demonstrate that, in the absence of original data, OOD-KA still offers an effective approach for customizing versatile student models based on pre-existing models. Our code is available at https://github.com/WuDiHJQ/OOD-KA.
Singing voice manipulations have become increasingly common in modern music production. While such techniques can serve as creative tools to enhance artists’ expressive possibilities, they can also raise concerns about content authenticity and media integrity. To counter potential misuse of vocal manipulation tools, recent research has developed detection systems. However, these are typically limited to binary classification, indicating only whether a vocal track has been altered, without providing any further interpretable information. In this work, we address this limitation and propose a novel framework that combines forensic audio analysis with natural language generation to both detect and describe modifications in singing voice signals. Building on recent advances in audio-language models, we construct a dataset of manipulated and synthetic vocals annotated with detailed textual annotations, which we use to train and evaluate our framework. Our approach identifies and characterizes a wide range of vocal transformations, including pitch correction, pitch shifting, time stretching, and singing voice deepfake generation. Experimental results show that the proposed method not only surpasses existing baselines in classification accuracy but also provides substantially greater interpretability, as it provides explanations of the outputs in natural language, making them understandable to non-experts. This makes the system particularly relevant for music production, media forensics, and copyright verification, offering a transparent and descriptive account of vocal alterations.
Recent advancements in CNNs and transformer-based methods have significantly improved multiclass anomaly detection; however, accurately localizing small anomalies in industrial images remains challenging. Although promising, CNNs suffer in capturing long-range dependencies and the effectiveness of the transformers is hindered by their quadratic computational costs. This paper proposes a novel State Space Model (SSMs)-based multi-class anomaly detection method, termed Pyramidal Anomaly Detection with State-Space Models (PAD-SSM), which offers state-of-the-art performance with linear complexity. The main contribution of our work is the Pyramidal Scanning Approach (PSA), which performs the scanning hierarchically. This enables more localized and scale-aware analysis. PSA operates in a pyramid-like style, recursively partitions the image into equal non-overlapping patches across multiple scales. The SSM is then applied independently to each patch. This approach enables detailed, multi-scale, and high-resolution analysis, which is crucial for detecting small anomalies in industrial images. We integrate the PSA block with a pre-trained encoder for multi-scale feature extraction, a feature adapter to transform the encoded features to a target domain, and a synthetic anomaly generator that introduces noise at the feature level, further enhancing detection capabilities. Rigorous evaluations on the MVTec-AD and VisA datasets demonstrate that our method achieves state-of-the-art mean AU-ROC of 98.7, AP of 99.6, and F1max of 98.1 on MVTec-AD benchmark while using the lowest number of FLOPS (e.g., 10.3% fewer FLOPs than the next best method).
Existing cross-dataset facial expression recognition (FER) approaches primarily rely on explicit feature alignment or discriminative transfer strategies, which often struggle to effectively model the underlying distribution and category-specific features in an unlabeled target domain. To address this issue, we propose a Domain-Category Fusion Guided Diffusion Model (DFGDM), aiming to enhance target domain adaptability from a generative modeling perspective. Specifically, a domain classifier based on Maximum Mean Discrepancy (MMD) is introduced to distinguish whether a sample originates from the target domain at a global level. Additionally, a category classifier constructed using Local Maximum Mean Discrepancy (LMMD) is employed to differentiate source and target samples from a category-specific perspective. Together, these two components provide domain-level and class-level discriminative guidance for the diffusion model, thereby strengthen its ability to generate samples that conform to the target domain’s feature distribution. Moreover, pseudo-labels are generated using a pretrained UDA model’s predictions on the target domain samples, which offer category supervision for the conditional diffusion model. This enables category-conditional sample generation in the absence of true target labels. The generated pseudo target-domain images are then merged with genuine source-domain samples to construct a hybrid source domain. Experiments are carried out on distinct datasets, namely FER2013, RAF, CK+, SFEW, SAMM and CASME II, involving seven groups of cross-domain facial expression recognition experiments under the unsupervised subdomain adaptation framework. The experimental results show that compared with SOTA, the proposed DFGDM method achieves optimal or comparable results, verifying the effectiveness of combining diffusion models with transfer learning in cross-dataset facial expression recognition (The corresponding code:https://github.com/yifancai1022/yue3).
In this paper, we introduce a new dataset for facial expression analysis that includes video clips extracted from movies, where two subjects interact with each other while showing a variety of emotions. Characterized by the joint modeling of Interpersonal relations and Multi-Emotions, we refer to our dataset as IMEmo for short. Overall, we collected 100 clips from romance and drama movies in different languages. The dataset consists of 79.3 min of scenes, with a duration of each clip ranging between 0.2 and 2.13 min. Each clip contains two people communicating verbally, through expressions, and via body pose and gestures. At the clip level, the dataset has been annotated with the age and gender of the two individuals as well as their social relationship. At a finer level of annotation, clips are split into shots that are in turn labeled with 16 fine-grained emotion categories, more than twice the number used in standard datasets that include 6 or 7 emotion labels. Emotion recognition in interpersonal relations is reported to also provide an estimation of the difficulty of the data in comparison to existing benchmarks for expression analysis. Baseline results are also given for the task of relation recognition and emotion prediction. In addition to introducing the IMEmo dataset, we provide detailed annotation protocols, expanded statistical analyses, new tasks such as interpersonal relation recognition and emotion prediction, and thorough benchmarking against multiple standard datasets for different methods. These additions mark a significant advancement in the modeling and understanding of interpersonal emotional dynamics in video.
Soft-biometrics triage aims to reduce a large gallery to a compact, defensible candidate set that is likely to contain the person of interest, using cues that are easy to observe (e.g., apparent sex, hair color, glasses, clothing, and accessories) but are not uniquely identifying on their own. We cast this task as coverage-controlled set selection from calibrated attribute predictions, instantiated by a shared encoder with family-specific heads and a conformal acceptance rule that admits gallery items whose attribute profiles are sufficiently consistent with the probe. We study two complementary regimes: portrait-style faces (CelebA) and surveillance-style pedestrians (PA-100K). For CelebA, calibration and evaluation use genuine identity pairs, where two distinct images of the same identity are sampled to estimate within-person variability. For PA-100K, identity annotations are unavailable; therefore, surrogate two-view pairs are constructed using independent light augmentations of the same image to approximate appearance variation. Our evaluation reports coverage, shortlist size, pool reduction, and probability calibration quality, and compares against workload-matched ranking baselines that lack inclusion guarantees. Across both datasets, the proposed triage substantially reduces gallery size while tracking user-specified coverage targets, with transparent rationales grounded in interpretable cue families (basics, local/changeable details, and accessories/patterns). We discuss limitations arising from surrogate views, label noise, attribute imbalance, and domain shift, and outline deployment considerations including recalibration and audit. The results indicate that calibrated, family-structured soft biometrics provide a practical and accountable basis for coverage-controlled forensic triage across domains.
The explosive growth in volume and increasing resolution in video data demand corresponding strategies for accurate identification. While existing methods predominantly rely on supplementary modules to boost performance, they overlook the potential of architectural optimization. This paper introduces CONet, a novel 2D Convolutional Neural Network (CNN) for video action recognition. Building upon the established paradigm of adapting 2D CNNs for video understanding, CONet introduces a systematic co-design approach that holistically optimizes both structural components and feature guidance mechanisms. It synergizes two core components: Linear Remix Generation (LRG), which refines the convolutional and pooling layers to capture low-frequency spatial contexts better and enable efficient parameter sharing across time; and Triple Adaptive Alignment (TAA), which approximates multi-head self-attention within a CNN framework to fuse high-frequency spatio-temporal semantics adaptively. This co-design enables CONet to effectively balance recognition accuracy and computational complexity. Extensive experiments demonstrate that CONet achieves state-of-the-art or competitive performance on major benchmarks (79.4% on HMDB51, 98.4% on UCF101, 81.1% on Kinetics400) while maintaining 30% fewer parameters than comparable Vision Transformers (ViTs). CONet also exhibits strong transferability across datasets and excels in real-world applications such as violence detection, underscoring its generalization capability beyond standard benchmarks.
