Partially Relevant Video Retrieval (PRVR) addresses the critical challenge of matching untrimmed videos with text queries describing only partial content. Existing methods suffer from geometric distortion in Euclidean space that sometimes misrepresents the intrinsic hierarchical structure of videos and overlooks certain hierarchical semantics, ultimately leading to suboptimal temporal modeling. To address this issue, we propose the first hyperbolic modeling framework for PRVR, namely HLFormer, which leverages hyperbolic space learning to compensate for the suboptimal hierarchical modeling capabilities of Euclidean space. Specifically, HLFormer integrates the Lorentz Attention Block and Euclidean Attention Block to encode video embeddings in hybrid spaces, using the Mean-Guided Adaptive Interaction Module to dynamically fuse features. Additionally, we introduce a Partial Order Preservation Loss to enforce “text ≺video” hierarchy through Lorentzian cone constraints. This approach further enhances cross-modal matching by reinforcing partial relevance between video content and text queries. Extensive experiments show that HLFormer outperforms state-of-the-art methods. Code is released at https://github.com/lijun2005/ICCV25-HLFormer.
Multi-task learning (MTL) trains deep neural networks to optimize several objectives simultaneously using a shared backbone, which leads to reduced computational costs, improved data efficiency, and enhanced performance through cross-task knowledge sharing. Although recent gradient manipulation techniques aim to find a common descent direction that benefits all tasks, conventional empirical loss minimization still leaves models vulnerable to overfitting and gradient conflicts. To address this, we introduce a novel MTL framework that leverages weight perturbation to regulate gradient norms, thus improving generalization. By carefully modulating weight perturbations, our approach harmonizes task-specific gradients, reducing conflicts and encouraging more robust learning across tasks. Theoretical insights reveal that controlling the gradient norm through weight perturbation directly contributes to better generalization. Extensive experiments across diverse applications demonstrate that our method significantly outperforms existing gradientbased MTL techniques in terms of task performance and overall model robustness.
The early stage of multi-modal pre-training plays a pivotal role in aligning two modalities for Large VisionLanguage Models (LVLMs), while evaluating its training quality usually requires the costly supervised fine-tuning (SFT) stage to verify the downstream benchmark scores. Loss, perplexity, and in-context evaluation results are commonly used pre-training metrics for Large Language Models (LLMs), while we observed that these metrics are less indicative when quantifying the pre-trained LVLMs. Due to the lack of proper metrics, the research of LVLMs in the multi-modal fusion stage is hindered greatly, including the training data choice, efficient module design, etc. In this paper, we first present Modality Integration Rate (MIR), an effective, robust, and generalized metric to indicate the multimodal alignment quality of LVLMs without SFT. This metric evaluates LVLM pre-training from the inter-modal distribution distance perspective, which is 1) Effective to represent the fusion quality and show a positive relation with the benchmark performance after SFT, 2) Robust toward different training/evaluation data, and 3) Generalize across training configurations and architecture choices. Complementing MIR, we further propose learnable Modality Calibration (MoCa), a lightweight module to narrow the modality gap at each language model layer during training. A series of experiments are conducted to explore the effectiveness of MIR and MoCa, demonstrating that MIR is highly indicative about training data selection, training strategy schedule, and architecture design to improve pre-training. The code is at: shikiw/Modality-Integration-Rate.
Text-to-image (T2I) models are widespread, but their limited safety guardrails expose end users to harmful content and potentially allow for model misuse. Current safety measures are typically limited to text-based filtering or concept removal strategies, able to remove just a few concepts from the model's generative capabilities. In this work, we introduce AlignGuard, a method for safety alignment of T2I models. We enable the application of Direct Preference Optimization (DPO) for safety purposes in T2I models by synthetically generating a dataset of harmful and safe imagetext pairs, which we call CoProV2. Using a custom DPO strategy and this dataset, we train safety experts, in the form of low-rank adaptation (LoRA) matrices, able to guide the generation process away from specific safety-related concepts. Then, we merge the experts into a single LoRA using a novel merging strategy for optimal scaling performance. This expert-based approach enables scalability, allowing us to remove $7 \times$ more harmful concepts from T2I models compared to baselines. AlignGuard consistently outperforms the state-of-the-art on many benchmarks and establishes new practices for safety alignment in T2I networks. Warning: this paper includes potentially offensive content.
Seam cutting has shown significant effectiveness in the composition phase of image stitching, particularly for scenarios involving parallax. However, conventional implementations typically position seam-cutting as a downstream process contingent upon successful image alignment. This approach inherently assumes the existence of locally aligned regions where visually plausible seams can be established. Current alignment methods frequently fail to satisfy this prerequisite in large parallax scenarios despite considerable research efforts dedicated to improving alignment accuracy. In this paper, we propose an alignment-compensation paradigm that dissociates seam quality from initial alignment accuracy by integrating a Local Patch Alignment Module (LPAM) into the seam-cutting pipeline. Concretely, given the aligned images with an estimated initial seam, our method first identifies low-quality pixels along the seam through a seam quality assessment, then performs localized SIFT-flow alignment on the critical patches enclosing these pixels. Finally, we recomposite the aligned patches using adaptive seam-cutting and merge them into the original aligned images to generate the final mosaic. Comprehensive experiments on large parallax stitching datasets demonstrate that LPAM significantly enhances stitching quality while maintaining computational efficiency. The code is available at https://github.com/tlliao/LPAM_seam-cutting.
Robot foundation models, particularly Vision-LanguageAction (VLA) models, have garnered significant attention for their ability to enhance robot policy learning, greatly improving robot's generalization and robustness. OpenAI's recent model, O1, showcased impressive capabilities in solving complex problems by utilizing extensive reasoning chains. This prompts an important question: can robot models achieve better performance in multi-task, complex environments by reviewing prior observations and then providing task-specific reasoning to guide action prediction? In this paper, we introduce Chain-of-Affordance (CoA-VLA), a novel approach to scaling robot models by incorporating reasoning in the format of sequential robot affordances to facilitate task completion. Specifically, we prompt the model to consider the following four types of affordances before taking action: (1) object affordance — what object to manipulate and where it is; (2) grasp affordance — the specific object part to grasp; (3) spatial affordance — the optimal space to place the object; and (4) movement affordance — the collision-free path for movement. We further transform each affordance into two prompting formats: visual affordance and textual affordance. We introduce a novel vision-language co-injection module that integrates this knowledge into the policy network. This allows the robot to leverage essential contextual information during action inference, resulting in improved precision and robustness. Our experiments demonstrate that CoA-VLA outperforms state-of-the-art robot foundation models, including OpenVLA and Octo, on a variety of tasks. Furthermore, CoA-VLA exhibits strong generalization capabilities, including recognizing unseen object poses, identifying free space, and avoiding obstacles in novel environments.
Generating realistic 3D outdoor scenes is essential for applications in autonomous driving, virtual reality, environmental science, and urban development. Traditional 3D generation approaches using single-layer diffusion methods can produce detailed scenes for individual objects but struggle with high-resolution, large-scale outdoor environments due to scalability limitations. Recent hierarchical diffusion models tackle this by progressively scaling up lowresolution scenes. However, they often sample fine details from pure noise rather than from the coarse scene, which limits the efficiency. We propose a novel cubeabsorb discrete diffusion (CADD) model, which employs low-resolution scenes as the base state in the diffusion process to generate fine details, eliminating the need to sample entirely from noise. Moreover, we introduce the Sparse Cube Diffusion Transformer (SCDT), a transformer-based model with a sparse cube attention operator, optimized for generating large-scale sparse voxel scenes. Our method demonstrates state-of-the-art performance on the CarlaSC and KITTI360 datasets, supported by qualitative visualizations and extensive ablation studies that highlight the impact of the CADD process and sparse cube attention operator on high-resolution 3D scene generation.
Block Face Imaging (BFI) is a high-resolution snapshot acquired during histological sectioning that captures the exposed tissue surface using tissue autofluorescence of lipids and NADPH. BFI offers up to five times greater white-gray matter contrast than traditional imaging, allowing clear visualization of white matter(WM) tracts. To enable accurate segmentation of WM tracts, we introduce a MedSAM-guided curriculum learning that progressively improves performance by training on increasingly complex samples. This approach takes advantage of both the anatomical precision of BFI and the generalizability of the medical foundation models. The resulting segmentations are manually validated by expert neuroscientists to ensure anatomical accuracy. Our method achieves an average Dice score of 0.9297, evaluated across 230 BFI slices with a spatial resolution of approximately 60 mu m, offering significantly enhanced structural clarity over conventional MRI-based approaches (500 mu m resolution) and outperforming state-of-the-art segmentation techniques. These findings demonstrate the effectiveness of our approach in leveraging high-resolution BFI data for accurate WM tract segmentation, highlighting its potential as a complementary tool alongside conventional neuroimaging techniques.
Diffusion models can be used to generate highly realistic likenesses of individuals when fine-tuned on a few personal images, raising serious concerns about unauthorized use and identity exploitation. In response, proactive defenses have been proposed by incorporating imperceptible protective perturbations into images to prevent unauthorized fine-tuning. Although these techniques are promising, their robustness under real-world conditions, such as common post-processing and adversarial purification, remains largely unexplored. In this work, we first evaluate the robustness of state-of-the-art protective methods with various post-processing and purification techniques. Moreover, to address the limitations of existing purification methods of removing protective perturbations along with natural high-frequency components, we propose FreqPure, a high-frequency-aware, diffusion-based purification approach that incorporates frequency consistency constraint to better preserve image fidelity during purification. It integrates a two-stage purification pipeline: 1) A reconstruction module that removes artifacts introduced by protection techniques. 2) A diffusion-based model that synthesizes high-frequency components, ensuring the output remains realistic. Extensive experiment results show that the proposed approach can purify perturbation on the image efficiently while preserving the natural high-frequency details of the images.
Accurate and robust tooth segmentation in 3D dental meshes is essential for various clinical and diagnostic applications, yet existing multi-class segmentation models often suffer from label confusion, especially in cases involving missing, crowded, or misaligned teeth. To address these challenges, we propose Render2Seg, a modular two-stage segmentation framework that decouples object detection from segmentation. In the first stage, 3D dental meshes are rendered into 2D occlusal-view images, where a YOLO-based detector localizes and classifies individual teeth. Directional landmarks are then detected to guide anatomically aligned 3D patch extraction. In the second stage, each extracted patch is independently segmented using a binary segmentation network, avoiding inter-tooth label ambiguity. Our method demonstrates strong segmentation performance on the public Teeth3DS dataset, achieving high accuracy across diverse tooth types. Moreover, Render2Seg shows superior generalizability and robustness, making it a promising solution for real-world clinical integration. We also discuss current limitations in detection reliability and propose directions for future improvement.
Early detection and precise segmentation of polyps in colonoscopy images are crucial for the timely diagnosis and treatment of colorectal cancer. In this work, we introduce QPolypNet, an innovative hybrid quantum-classical deep learning architecture tailored for medical image segmentation. To the best of our knowledge, this is the first approach to apply quantum-enhanced techniques to polyp segmentation. QPolypNet integrates parameterized quantum circuits as adaptive feature enhancers with a ResNet-50 backbone, a feature pyramid network, and attention mechanisms to boost segmentation performance. Evaluated on four public datasets: CVC-ClinicDB, Kvasir-SEG, CVC-Colon-DB, and ETIS-Larib, our method achieves consistently strong performance across all datasets. Specifically, it yields Dice scores of 0.942 (CVC-ClinicDB), 0.892 (Kvasir-SEG), 0.911 (CVC-Colon-DB), and 0.932 (ETIS-Larib), surpassing state-of-the-art polyp segmentation models. These results highlight the potential of hybrid quantum-classical architectures to enhance medical image analysis and pave the way for further research into quantum-assisted healthcare solutions.
Language confusion, manifested as unintended interlingual mixing and inconsistent linguistic outputs, significantly impairs the effectiveness and reliability of multilingual Large Language Models (LLMs) and Large Multimodal Models (LMMs), particularly affecting those processing non-Latin scripts. This paper introduces the Confusion-Aware Preference Optimization Pipeline, which integrates innovative Seen Token Allowances and advanced Named Entity Recognition (NER) Allowances designed specifically to identify and regulate language-specific entities. These components collaboratively enhance both token-level and entity-level language control within Direct Preference Optimization (DPO) training, effectively mitigating language confusion. Additionally, we propose a novel evaluation metric that explicitly quantifies language confusion through comprehensive Seen Token and NER analyzes, incorporating GPT-driven LLM-as-a-Judge methods for linguistic quality assurance and early stopping. Empirical evaluations confirm the effectiveness of our proposed pipeline, demonstrating significant improvements in multilingual coherence and linguistic accuracy. As the first dedicated exploration addressing language confusion within Vision-Language Models (VLMs), this research represents a critical advancement towards developing robust and scalable multimodal foundation models with reliable multilingual capabilities.
Vision-Language Models (VLMs) have made significant progress, yet they still struggle with fine- grained spatial understanding at the region level, especially in complex environments such as warehouses, due to limited spatial perception and reliance on raw input data. Existing methods often represent region information textually, overburdening the LLM's reasoning and failing to fully exploit depth information for detailed spatial relationships. To address this, we propose an enhanced SpatialRGPT architecture that integrates a novel Region Enhancer module to strengthen interaction and encoding between RGB and depth features at the region level. We also incorporate a specialized Region Classifier to perform direct object classification on specific regions, reducing the load on the LLM and improving accuracy. Our multi-stage training paradigm effectively fine-tunes these new modules while leveraging transfer learning. Experiments on the PhysicalAI Spatial Intelligence Warehouse dataset (AI City Challenge 2025, Track 3) show substantial improvements on spatial reasoning tasks over the baseline. The overall performance consistently improves across training phases, validating the effectiveness of our enhanced region representations, specialized classification module, and the staged training procedure.
Apples are typically harvested once a year and stored under controlled atmosphere conditions to preserve quality. However, internal defects such as internal browning and cavities can develop during storage. X-ray phase contrast imaging, which simultaneously captures transmission and dark field modalities, has emerged as a promising non-destructive technique for internal quality assessment. These two complementary modalities require effective fusion strategies to be fully leveraged in automated inspection systems. This study investigates three fusion strategies, namely pixel-level, feature-level, and decision-level, combined with multiple feature extraction methods and classifiers for binary classification of healthy and defective apples using X-ray dark field and transmission images. Models were evaluated on two test sets from different seasons, storage conditions and durations. Feature-level fusion using DINO ViT-B/8 feature extractor and logistic regression achieved the highest performance, reaching a balanced accuracy of 0.92 on the within-distribution test set and 0.94 for another independent test set. The findings highlight the importance of combining robust feature representations with multimodal fusion to improve classification performance and generalization. The proposed framework could be extended to other horticultural products exhibiting internal quality disorders.
Document Visual Question Answering (DocVQA) offers a promising approach to extracting insights from large document corpora. However, existing benchmarks focus on evaluating multi-modal understanding within a single document. This gap hinders the development of methods integrating scattered information across pages and documents. To address this, we introduce M3Doc VQA, the first benchmark designed for multi-modal, multi-page, and multi-document understanding. M3DocVQA comprises over 3,000 PDF documents with more than 40,000 pages, offering a challenging environment where evidence is distributed across diverse sources and modalities. Along-side the dataset, we introduce M3DocRAG, a baseline method based on multi-modal retrieval-augmented generation. M3DocRAG flexibly handles both single and multiple document settings while preserving critical visual information, establishing a useful starting point for future work in open-domain multi-modal document understanding. Our experiments across three benchmarks (M3DocVQA, MMLongBench-Doc, and MP-DocVQA) show that existing methods struggle with open-domain question answering over extensive, multi-modal documents. Although M3DocRAG has shown promising performance, there is large room for future improvement. We provide comprehensive ablation studies of different indexing, multi-modal language models, and multi-modal retrieval models, along with qualitative examples to guide future research.
While Vision-Language Models (VLMs) offer transformative potential for cultural heritage preservation, they often exhibit significant “cultural blind spots” due to training data heavily skewed towards western contexts. This leads to limited understanding of non-western cultures, such as those from East Asia. This paper posits that modern VLMs consequently fail to accurately interpret underrepresented cultural objects, leading to misidentification, cultural confusion, and factual hallucination. To investigate this, we evaluate prominent VLMs including LLaVA-1.5, ViP-LLaVA, Shikra, and MiniGPT-4 on a newly curated, culturally-rich Visual Question Answering (VQA) dataset specifically focused on traditional Korean attire, Hanbok. Our experimental results demonstrate that these models not only exhibit low accuracy but also reveal systematic error patterns indicative of a deeper lack of cultural understanding. Beyond diagnosing this deficiency, we propose a methodological refinement through the adoption of ‘thick’ evaluation frameworks that move beyond superficial accuracy metrics, explicitly assessing nuanced cultural understanding and alignment. Furthermore, we propose Multi-modal Retrieval-Augmented Generation (MRAG) as an enhanced architectural paradigm to ground models explicitly in verifiable, culturally contextualized; and communitycurated knowledge, addressing fundamental shortcomings of existing methods. This work provides empirical evidence of cultural limitations inherent in current VLMs and charts a research agenda toward building more equitable and culturally respectful AI for global digital heritage.
Visual explanation methods have been effective in interpreting the outputs of object detectors by highlighting important regions corresponding to each model prediction. However, existing approaches have largely overlooked inter-object relationships-particularly the relative importance of each pixel across different objects, a concept we refer to as Object Discrimination (OD). In this paper, we propose difference maps, a novel visual explanation technique designed to enhance the interpretability of object detectors with respect to OD. Serving as a complementary tool to existing instance-specific heat maps, difference maps improve their ability to isolate the impact of key features of individual objects on model outputs. Our qualitative and quantitative evaluation results show that the proposed difference maps can effectively distinguish key features specific to the target object, capturing the relative importance of each pixel across different predictions within the same scene. Our method is applicable to a wide range of object detectors, including one-stage, two-stage, and transformer-based architectures. Furthermore, it enhances existing heatmap-based visual explanation methods by improving focus on the detected object. These results demonstrate the utility of our approach in improving model transparency and inter-pretability across different detection architectures and explanation techniques.
Adversarial attacks have emerged as a critical threat vector against deep learning models, enabling the manipulation of model outputs through carefully crafted input perturbations. In this study, we explore the effectiveness of three black-box adversarial attack methodologies-where the attacker has no access to model internals-in the context of skin image classification. Using clinically relevant dermatological image datasets, we demonstrate that these attacks can reliably degrade the predictive performance of state-of-the-art convolutional neural networks, even under strict black-box constraints. We evaluate the proposed approach using black-box, noise-driven adversarial perturbations, including uniform random noise, Gaussian-masked noise, and anatomically-constrained noise guided by segmentation masks generated via the Segment Anything v2 (SAM2) model. The proposed black-box attacks resulted in a significant reduction in model accuracy (up to 23.75%) and F1-Score (as much as 50% drop). These results reveal a critical vulnerability of medical image classifiers to adversarial attacks and the importance of incorporating robust safety mechanisms during model development and deployment. The proposed models employ transfer learning using architectures such as ResNet-50 and EfficientNet (B0,B4, and B7). Training is conducted in two phases: initially, the backbone is frozen while a custom classification head is trained to adapt to the skin imaging domain. Subsequently, a two-stage unfreezing strategy is applied, grad-ually fine-tuning deeper layers of the network to improve task-specific feature representation. All models are trained on ISIC archive datasets, utilizing the 3D whole-body scans from ISIC 2024 and dermoscopic images from ISIC 2018 to ensure diversity in acquisition modalities and improve generalization across clinically relevant input types.
Extracting information from documents is a critical task for many industrial use-cases. Errors in this process can arise from both the visual recognition and the semantic labeling processes. Traditional approaches to mitigate these errors involve collecting more data and training larger models, which can be resource-intensive. In this paper, we propose a backtracking, constrained decoding approach that aims to correct OCR reading and information extraction at the inference stage without retraining or using additional error correction models. By leveraging OCR confidence scores and top-k predictions, we explore multiple high-probability OCR readings until we reach a constraint-satisfying extraction. Our approach uses computational resources during inference to jointly solve both OCR and extraction issues. We demonstrate the effectiveness of our method on two datasets, showing consistent improvements in F1-score, compared to degradation in extraction performance when using a state-of-the-art post-OCR error correction model specifically fine-tuned on these datasets.
This paper presents an overview of the AIM 2025 Challenge on Screen Content Video Quality Assessment. The challenge included a set of 150 source videos. To receive distorted versions, the source videos were transmitted through video conferencing applications, introducing real-world distortions such as compression artifacts and frame drops. Distorted versions were labeled by human crowdsourcing assessors to receive reference subjective scores. The evaluation was based on subjective quality assessment via crowdsourcing, obtaining votes from over 8,000 assessors. The goal of the participants was to develop an algorithm to assess the visual quality of the videos, achieving the highest correlation with the subjective scores. The challenge attracted more than 45 registered teams, 5 of which passed the final phase with source code verification. The outcomes may provide insights into the state of the art in screen-content video quality assessment and highlight emerging trends and effective strategies in this evolving research area. All data, including the processed videos and subjective comparison votes and scores, is made publicly available - https://github.com/msu-video-group/AIM25_SC_Quality_Assessment
