Video customization has gained significant interest, yet achieving temporal and geometric consistency remains challenging. To address this, we propose a novel framework that leverages neural representations to model dynamic 3D scenes in videos, distilling their knowledge into rendering consistency to regularize diffusion models. Our method conceptualizes dynamic scenes as 4D volumes, employing grid-based dynamic geometry representations to enhance geometric consistency and cross-view appearance representations to improve visual coherence across frames. To seamlessly integrate these representations into diffusion models, we propose a latent neural rendering strategy that aligns geometry and appearance in a unified manner. Comprehensive evaluations across tasks such as local object editing and composite customization reveal consistent and significant performance gains, demonstrating the effectiveness and robustness of our proposed method.
The effective modeling and querying of the world’s ever-growing collection of multimedia is a significant challenge. Multimodal knowledge graphs offer a powerful solution by integrating heterogeneous data sources and uncovering complex relationships. We point to its benefits across a range of applications, including holistic memory retrieval, interactive video exploration, cross-modal fact-checking for journalism, and data fusion for clinical diagnosis. We also explore its use in enhancing educational tools, product feedback analysis in e-commerce, and semantic annotation of textual corpora. By showcasing how these diverse applications model multimedia at its core, we illustrate the potential of a multimodal knowledge graph store that can directly leverage and process the content of multimedia. However, a major limitation of existing approaches is that they treat multimedia documents as opaque entities, which severely limits their analytical potential. In this paper, we introduce MeGraS, our novel multimodal knowledge graph store that embodies a new paradigm: to provide direct access to multimedia document content, enabling deeper and more flexible analysis.
No-Reference Image Quality Assessment (NR-IQA) methods has made significant progress with the rapid development of deep learning techniques. However, most NR-IQA methods usually use a single attention mechanism for feature interaction, which limits the prediction performances of these NR-IQA methods in complex scenarios. To address this, we propose a Feature Enhancement and Feature Interaction Network (FEFI-Net) for NR-IQA. The key contributions are the multi-scale attention enhancement module and the multi-attention feature interaction module. The multi-scale attention enhancement module is designed to enhance the pre-trained features extracted by Swin Transformer, thereby obtaining more comprehensive feature representations. The multi-attention feature interaction module with three attentions is designed to establish more accurate feature interaction. Comparative experiments on five widely-used IQA datasets demonstrate that our FEFI-Net surpasses some state-of-the-art NR-IQA methods in prediction accuracy.
The rapid dissemination of multimodal content has intensified the spread of fabricated news, presenting a substantial threat to social integrity. A formidable challenge for current detection systems is identifying misinformation related to novel events in zero-shot scenarios. Prevailing zero-shot methods typically assess news items in isolation via semantic matching, a strategy that fails to recognize the recycled disinformation tactics from past campaigns and lacks the sophisticated reasoning needed to identify subtle, cross -modal discrepancies. To surmount these deficiencies, we introduce MRAFnd, a novel Multimodal Retrieval-Augmented Framework for Zero-Shot Fake News Detection. MRAFnd emulates a collaborative team of analysts to verify news veracity. The framework initiates with Multimodal Similarity-based News Retrieval to assemble a corpus of contextually analogous articles from an unlabeled reference database. Subsequently, during the Bifurcated Evidential Reasoning stage, agents perform a dual -directional analysis to extract critical patterns from the retrieved evidence. Finally, a Multi-Agent Collaborative Debate, involving Analyst and Arbiter agents, engages in a structured discourse to arrive at a definitive and robust conclusion. Comprehensive experiments on three benchmark datasets reveal that MRAFnd markedly surpasses state-of-the-art baselines, achieving an accuracy gain of up to 2.35% on the demanding Weibo-21 dataset.
Language-guided video object segmentation (LVOS) has achieved remarkable progress through advanced video segmentation models, but their performance heavily relies on large-scale fine-grained training data. The acquisition and fine-grained annotation of video datasets require exorbitant labeling costs and considerable time overhead. To address this challenge, we present DiffSynth-LVOS, a novel method that contains a diffusion-based object video generation model to create synthetic various training data according to textual object description. Our method first generates high-quality videos conditioned on masks from existing real datasets by introducing a full-fill mask constraint and video latent blending module, ensuring precise frame-mask alignment and temporal consistency for video segmentation training. Then the impact of synthetic videos through joint fine-tuning with real videos is investigated to increase the performance of LVOS. Experimental results demonstrate significant performance improvements when training on our expanded dataset (e.g., 𝒥&ℱ score improves from 52.39
We present an efficient method to distill reasoning capabilities into compact video-language models (VLMs) for video question answering (VideoQA). Our approach fine-tunes a 2B-parameter model using only ∼ 900 uncertainty-selected examples, each augmented with synthetic chain-of-thought (CoT) rationales generated by a 4B teacher. Despite its minimal compute cost–under two hours on a single A100 GPU–our method enables the 2B model to outperform VLMs up to 4 × larger, and generalize across CinePile, ActivityNet-QA, and MLVU, approaching the performance of its own 4B teacher. A key finding is that placing CoT rationales after the answer–contrary to standard prompting–substantially improves reasoning in compact models. This insight challenges prevailing CoT conventions and reveals new alignment strategies under limited model capacity. Our findings offer a practical blueprint for training deployable, reasoning-rich VLMs suited for mobile and edge applications.
In recent years, frequency dynamic convolution (FDConv) has shown strong performance in sound event detection (SED). However, it primarily focuses on frequency modeling, neglecting dynamic temporal variations and time-frequency interactions, which limits its ability to capture key patterns in non-stationary events. To address this, we propose a time-frequency dual-stream attention (TFDSA) mechanism that integrates temporal attention pooling (TAP) and spectral feature booster (SFB) into conventional attention, enabling separate modeling of non-stationary and stationary structures. TFDSA employs a tri-branch design, including differential perception, gated modulation, and statistical smoothing, to enhance features and provide saliency guidance along both temporal and spectral dimensions. Experiments on the DESED dataset demonstrate that TFDSA outperforms the FDConv baseline, improving PSDS1 and PSDS2 by 2.53
The immense volume of user-generated video content demands scalable summarization methods that do not require costly human annotations. Unsupervised approaches provide this flexibility but still face key limitations: reliance on unimodal inputs, weak temporal modeling, and deterministic outputs that fail to capture uncertainty. To overcome these limitations, we propose MBSum, a novel unsupervised multimodal video summarization framework that leverages both visual and textual modalities without requiring labeled data. MBSum combines a Mamba-based state space backbone for efficient long-range temporal modeling with Bayesian variational decoder for a robust, uncertainty-aware summary generation. In addition, we propose a text-guided contrastive loss to align visual and textual features, enhancing cross-modal coherence. MBSum supports both unimodal and multimodal inputs, which makes it adaptable to diverse real-world scenarios. Extensive experiments on five benchmarks (TVSum, SumMe, Soccer, MLB, and LoL) demonstrate the effectiveness of MBSum, achieving state-of-the-art performances on all datasets.
Neural Radiance Fields (NeRF) have demonstrated remarkable capabilities in high-fidelity 3D scene reconstruction. However, existing methods face challenges in balancing computational efficiency with anti-aliasing performance. While TensoRF improves training efficiency by leveraging explicit voxel grids and tensor decomposition, it fails to handle multi-scale data, leading to aliasing artifacts. In contrast, Mip-NeRF addresses the anti-aliasing issue through cone sampling and integrated positional encodings but relies on an implicit MLP-based representation, which is incompatible with explicit voxel grids. To solve these problems, this paper proposed RC-NeRF, a NeRF framework that incorporates a new sampling strategy combining both cone and ray sampling within explicit voxel grids, to achieve a balance between efficiency and robust anti-aliasing. RC-NeRF introduces learnable weights to adaptively associate the outputs of cone and ray sampling, and incorporates a regularization term to prevent the model from converging to a local optimum. The experimental results show that the proposed RC-NeRF not only improves performance in multi-scale view synthesis but also suppresses floating artifacts introduced by input real-world images.
Accurate medical image segmentation is essential for reliable clinical diagnosis, treatment planning, and outcome evaluation. However, existing transformer- and CNN-based approaches often fail to jointly capture long-range contextual dependencies and emphasize task-relevant structures, limiting their ability to delineate complex anatomical boundaries. To address these issues, we propose CCASNet, a dual-channel spatial modeling framework that unifies global structural reasoning and fine-grained representation refinement. Specifically, CCASNet introduces a dual-channel complementary attention mechanism, in which (i) a Criss-Cross Attention branch captures long-range dependencies and ensures global structural coherence with reduced complexity, while (ii) a Channel-Spatial Attention branch adaptively emphasizes diagnostically relevant regions and suppresses background noise. Unlike previous works that simply apply attention in a single dimension, our design explicitly models global – local complementarity, enabling robust delineation of complex anatomical boundaries. This complementary design enables CCASNet to simultaneously achieve global consistency and local precision. Extensive evaluations on Synapse and ACDC datasets demonstrate that CCASNet consistently outperforms competitive baselines, achieving an average Dice score of 85.62
Low-light images often suffer from multiple intertwined degradation factors, leading to poor visual quality. Although recent low-light image enhancement (LLIE) models have achieved notable progress, they still face challenges in effectively disentangling these factors while balancing enhancement accuracy and computational efficiency. In this work, we propose an illumination-prior guided hybrid network that integrates the strengths of global feature extraction (Transformer and Mamba) and local feature extraction (CNN). To facilitate effective feature fusion, we exploit illumination priors directly derived from the input images through a brightness-aware dynamic gating mechanism. Along the main encoder – decoder pathway, a dynamic window strategy is employed to preserve multi-scale perception while keeping the model size compact. Furthermore, a wavelet-based refinement module is introduced to separately restore high-frequency textures and low-frequency illumination, further improving the final output quality. Extensive experiments on multiple LLIE benchmarks demonstrate that our method outperforms SOTA methods in enhancing performance with acceptable computational costs. Ablation studies further confirm the effectiveness of each key component.
Diffusion models have recently demonstrated strong generative capabilities, yet their potential in unsupervised image clustering remains underexplored. In this work, we propose Diffusion-driven Representation Decoupling Clustering (Diff-RDC), which reconsiders the diffusion probabilistic modeling of denoising guided jointly by semantic features and latent cluster assignments. We perform discriminative semantic inference over the entire diffusion trajectory and eliminate timestep sampling during optimization, leading to a more principled Evidence Lower Bound (ELBO) that reinforces cluster separability while preserving class-conditional generation. To implement this formulation, we design an asymmetric encoder architecture that separates discriminative and fine-grained representations, where the former initiates semantic reconstruction and the latter progressively injects fine-grained details during denoising. Without relying on explicit disentanglement constraints, our framework implicitly achieves representation decoupling through a structured decoding flow. Extensive experiments on four diverse benchmarks validate the effectiveness of our method in improving clustering performance and semantic consistency. These findings are further corroborated by ablation studies and generative visualizations.
Vision Transformers have recently demonstrated strong capabilities in modeling the global context for camouflaged object detection. However, existing transformer-based approaches still struggle in complex scenarios involving small, multiple, and occluded objects. Their manually designed attention mechanisms often fail to adapt to data-specific characteristics, thereby limiting the model’s ability to capture long-range dependencies. Moreover, many of these methods are insufficient in capturing fine-grained local details. To address these limitations, we present a novel framework named SHNet. Specifically, we propose a spectral bias injection module (SBIM), which injects spectral deviation signals into the standard convolutional pathway to enhance the detection of small camouflaged objects. Then, we design a frequency attention module (FAM) based on the HiLo attention mechanism. By jointly modeling hierarchical dependencies across global and local contexts it improves performance in scenes with multiple and occluded objects. Furthermore, we design a plug-and-play interactive fusion module (IFM) that adaptively performs fine-grained feature selection and aggregates complementary information across different levels. Extensive experiments on four widely used datasets demonstrate the effectiveness and efficiency of the proposed method.
Deep neural networks (DNNs) are vulnerable to adversarial attacks, which inject subtle perturbations into input data, leading to misclassifications. Detecting adversarial examples before classification can effectively mitigate the risk of misclassification. However, existing detection methods typically employ global reconstruction with a single strategy, which not only disrupts clean regions but also fails to adapt to varying image complexity, often overprocessing simple areas while underprocessing complex ones, ultimately reducing detection accuracy. To address these issues, we propose a novel detection approach using image-adaptive local reconstruction (RIDA). Instead of applying uniform global reconstruction, RIDA selectively reconstructs classification-relevant regions and dynamically adjusts the reconstruction strategy based on local complexity: smooth regions are denoised with lightweight filtering, while structurally complex areas undergo refined restoration using deep residual modeling. This targeted reconstruction enhances the difference between pre- and post-reconstruction predictions, which are concatenated and classified by a lightweight SVM to detect adversarial examples. Experimental results on CIFAR-10 and ImageNet demonstrate that RIDA significantly improves adversarial example detection accuracy over baseline methods across diverse attack types.
In recommending songs to users, various types of relationships can be considered, such as songs liked by users with similar preferences or songs that are acoustically similar to those the target user already likes. Providing explanations for recommendations based on such relationships improves transparency and trust, but users currently have no control over which relationships are emphasized. To solve this problem, we extend an existing recommendation method based on a graph convolutional network (GCN) by representing each relationship as a separate graph layer with adjustable weights. By applying this method, we implemented a song recommender system with three types of relationships (user preference similarity, acoustic similarity, and creator commonality) on a music web service called “Kiite.” On the service, four types of recommendation results are displayed, depending on which relationships are emphasized and to what degree. The recommender system offers both transparency and controllability in that users can freely switch between the four recommendation result types. An analysis of over two years of usage logs demonstrates the effectiveness of combining transparency and controllability in music recommendation.
Recommendation systems (RS) have been extensively studied in recent years. This work focuses specifically on rating prediction, with the aim of estimating a product’s rating score from user reviews and ratings. Existing methods focused on textual reviews and user feedback to extract more information to address data sparsity and the cold-start problem. However, outliers significantly affected these methods and struggled to effectively capture the full meaning of textual data. To tackle this challenge, in this paper, we introduce a comprehensive data preprocessing method to handle outlier data before model input. Furthermore, addressing the limitations of previous LDA-based topic modeling approaches, we propose a novel model called PreBERT-Rec that leverages BERT representations with unsupervised clustering techniques (e.g., KMeans, DBSCAN...) as a powerful tool for topic modeling, enabling more effective information extraction. Extensive experiments on Amazon benchmark datasets demonstrate the superiority of our proposed method, achieving up to a 27.63
Deep Neural Networks (DNNs) exhibit notable susceptibility to adversarial attacks, which introduces substantial risks in safety-sensitive applications like self-driving vehicles and face identification. Unlike traditional targeted 2D attacks, targeted 3D adversarial attacks are more practical in real-world scenarios due to their effectiveness from multiple viewpoints. However, the transferability of adversarial examples generated by existing targeted 3D adversarial attack methods across different architectural models remains relatively limited. These methods inadequately capture the gradient differences among surrogate models, lack structured perturbation optimization, and fail to incorporate multi-view robustness constraints, thereby hindering the generalization of adversarial examples to unknown models and diverse physical conditions. To address these issues, we propose AIT3D-DSR, a novel method combining model integration with differentiable structured rendering. AIT3D-DSR employs a dynamic multi-model gradient integration strategy to align decision boundaries and incorporates block-level geometry-texture joint perturbations with physics-aware noise injection and randomized viewpoint transformations, thereby enhancing multi-view robustness. Comprehensive experiments show that AIT3D-DSR enhances the attack success rate by 27% compared to the best baseline, with an average success rate of 86.80% . It also demonstrates superior adversarial naturalness, as evidenced by a high SSIM value of 0.9002, a PSNR value of 30.59 dB, and a low LPIPS score of 0.0819. These results highlight AIT3D-DSR’s effectiveness in improving transferability while maintaining visual naturalness.
Multi-modal object detection has emerged as a critical research direction in recent years, as complementary characteristics from different modalities can effectively enhance detection accuracy and model robustness. This study proposes an innovative approach by investigating feature representation mechanisms through dual perspectives of temporal and frequency domains. In the frequency domain, differentiated filtering mechanisms are implemented to separately extract high-frequency edge information from infrared images and textural detail features from visible light images. The disentangled multi-modal features are subsequently integrated through a hierarchical fusion framework. Comprehensive experiments on the public multi-modal infrared-visible DroneVehicle dataset demonstrate the superior performance of our algorithm in small target detection tasks. The proposed method achieves state-of-the-art performance (mAP = 84.6
This paper investigates the problem of human motion sequence generation conditioned on action labels. Generating realistic motion sequences is of great significance for enhancing the performance of vision-related tasks such as action recognition and pose analysis. However, existing methods still suffer from limitations in generation accuracy and effective utilization of conditional information. In particular, the VQ-VAE architecture has restricted capability in modeling global features of motion sequences and insufficient exploitation of conditional cues. To address these issues, we propose a novel VQ-VAE architecture that learns high-quality discrete representations. The proposed framework leverages the complementary strengths of Transformers and CNNs, while integrating conditional information effectively during the decoding stage to improve generation quality. Furthermore, we introduce a generative pre-trained autoregressive model to predict discrete codebook indices, thereby further enhancing the quality of sequence generation. By integrating these two modules, we develop a generative model that produces human motion sequences conditioned on action labels, and conduct systematic evaluations on the HumanAct12 and UESTC datasets. Experimental results demonstrate that our method significantly outperforms several existing approaches in generation quality, validating the competitiveness and potential of the proposed VQ-VAE framework for human motion sequence generation.
Multimodal image matching establishes accurate correspondences between images captured by different sensors or imaging modalities. However, this task poses significant challenges due to substantial variations in radiometric properties, structural patterns, and texture distributions across modalities. To address these issues, this paper proposes FFMatch, a FilterFormer-based network designed for accurate multimodal image matching. Specifically, the Multi-Receptive Field Feature Aggregation module is introduced to enhance the model’s perception of multi-scale structural information and its capability for context modeling. The Token Filtering module is designed to dynamically model the spatial importance of features, enabling effective compression and filtering of redundant tokens. The Cross-Modal Alignment module is constructed based on a dual-stage cross-attention mechanism to improve structural consistency and information exchange across modalities. Furthermore, the Multi-Expert Fusion module is incorporated to enhance the model’s adaptability and discriminative ability in handling non-rigid deformations and local texture variations through multi-path collaborative modeling. Extensive experiments on several representative benchmark datasets demonstrate that FFMatch outperforms existing methods regarding matching accuracy, robustness, and generalization capability.
