Multiple Object Tracking (MOT) in dynamic and densely populated scenes presents significant challenges due to frequent occlusions, erratic object motion, and identity switches. While recent Transformer-based approaches have successfully leveraged global attention for object detection, most rely on temporal reasoning at the decoder level, leaving encoder-stage modeling underexplored. In this work, we propose an encoder-level temporal reasoning Transformer framework that embeds historical object trajectory information into the encoder stage via a tracklet memory. The encoder module, enhanced by Attention-by-Tracking, enriches visual features with temporal priors, while the decoder leverages Tracking-by-Attention to guide identity association using learned tracklet representations. To further improve temporal consistency and object localization, we introduce a gating-based temporal feature fusion mechanism that adaptively integrates multi-frame features based on cosine similarity. Our method is model-agnostic and can be applied to various Transformer-based MOT frameworks. When integrated into existing models such as TransTrack, MeMOTR, and MOTIP, it yields consistent performance improvements. Extensive experiments on DanceTrack and SportsMOT benchmarks demonstrate that our approach achieves superior tracking performance, including a HOTA score of 76.4 on SportsMOT. These results validate the effectiveness of encoder-level temporal integration and adaptive feature fusion for robust multi-object tracking in real-world scenarios.
Image color enhancement is a key component of ISP pipelines, with 3D LUTs widely adopted for nonlinear color transformations and color refinement. Existing learning-based approaches attempt to learn a global mapping from paired pixel-aligned training samples, but this offers limited control and flexibility for practitioners and restricts adaptation to specific images or scenarios. Moreover, the assumption of pixel-aligned supervision requires costly manual retouching to generate ground-truth references. A practical alternative is to use DSLR images as high-quality references, but the inevitable FoV and PoV variations make pixel-level 3D LUT calibration unreliable. We propose ChameleonTuner1, a novel framework that establishes region-level color correspondences to handle FoV/PoV discrepancies. Unlike prior learning-based 3D LUT methods, the proposed framework defines a dedicated search space for LUT tuning and employs multi-objective search to achieve controllable and interpretable optimization. Extensive experiments across multiple perceptual color metrics demonstrate that our method is effective in subjective scenarios (e.g., DPED) with geometric misalignments, while remaining competitive on standard pixel-aligned benchmarks (e.g., MIT-Adobe FiveK). Our code is available at https://github.com/ZjTan4/ChameleonTuner.git.
Generative artificial intelligence has the potential to improve productivity and transform the production of creative content. However, existing research indicates that image generation models are significantly influenced by biases. This work investigates the inherent biases and language-induced biases present in text-to-image models within the context of occupation-related image generation, complementing established metrics with human preference feedback. We present a comprehensive evaluation of five current text-to-image models: Midjourney v6.1, Stable Diffusion 3 Medium, DALL-E 3, Playground v2.5, and FLUX.1-dev , focusing on gender and ethnicity bias, image quality, and prompt alignment. To facilitate this evaluation, we developed the "Battle-Arena for Fair Image Synthesis" (BAFIS), a platform designed to collect human feedback on bias in generated images. Furthermore, we created a dataset comprising 21,140 synthetic images generated using multilingual prompts, which serves as a basis for our analysis. We further place our results within a broader social context by comparing them to official statistics from the German Federal Employment Agency. Our findings reveal systematic biases in text-to-image models, with established evaluation metrics in partial correlation with subjective user ratings. Thus, our research emphasizes the need for including human preferences to develop fairer and more inclusive text-to-image models.
The advanced multimodal processing of current vision language models (VLMs) has prompted rigorous benchmarking across multicultural settings, revealing a clear inclination toward Western culture. While the bias likely stems from the predominance of Western-centric images in the VLM pretraining data, the resulting long-tail distribution problem is only exacerbated in underrepresented cultural settings, such as Bengali. Our work explores this problem through an aspect-based evaluation of several classes of VLMs on the rich Bengali culture. Our BanglaProtha dataset is a VQA dataset, containing images that encapsulate Bengali cultural elements, questions in native Bengali, and semantically similar multiple-choice answer options. Our experiments provide behavioral insights into VLMs across prompting & fine-tuning strategies, cultural aspects, model size, and augmentation methods. Our work serves as a diagnostic tool for addressing and mitigating inequalities in multicultural and multilingual settings, thereby bringing efforts to democratize AI systems. Our code and data are available at https://github.com/farhanishmam/BanglaProtha.
Generating high-quality sketches from video requires a nuanced understanding of semantic content and visual structure, particularly for complex scenes across diverse sketch styles. Efficient and flexible video-to-sketch style transformation remains a significant challenge. We introduce VideoSketcher, a training-free framework for style-controllable sketch video generation that preserves frame structure while applying specified sketch aesthetics. Leveraging text-to-image diffusion models, VideoSketcher utilizes strong semantic priors without the need for extensive training. Our approach enforces temporal consistency by retaining latent information across frames and employs a Time-Linked Attention mechanism to capture structural elements from the source video and inject stylistic information from the reference image. To bridge the semantic gap between sketches and original video content, we introduce Sketch Directive Amplification for selective transfer of stylistic features. Additionally, a Stroke Graph Regularization strategy, comprising line and point loss, refines line consistency in the latent space. Extensive experiments validate VideoSketcher’s superior temporal stability and fidelity across diverse sketch styles and content. Video demos can be found in the supplementary materials.
The growing scale and heterogeneity of remote sensing (RS) imagery demand robust, scalable frameworks for content-based image retrieval across sensor modalities. We introduce X-JEPA, a novel predictive self-supervised architecture explicitly designed for cross-modal remote sensing image retrieval (RS-CMIR), and the first to extend joint embedding predictive paradigms beyond unimodal domains. Unlike prior contrastive or reconstruction-based methods, X-JEPA formulates representation learning as a latent forecasting task: predicting the semantic embedding of a target modality given context from another. To enforce modality-invariant alignment, we propose a geometry-aware Prediction Space Alignment (PSA) loss, which captures the structure of the latent space without requiring pixel-level reconstruction or modality pairing. We evaluate X-JEPA on two large-scale benchmarks—BEN-14K (Sentinel-1/Sentinel-2) and fMoW (RGB/Sentinel) across both unimodal and cross-modal retrieval tasks. X-JEPA consistently outperforms state-of-the-art self-supervised baselines, including MAE, SatMAE, CrossMAE, CSMAE-SESD, CROMA, SkySense, DeCUR, and REJEPA, achieving up to 11.0% F1-score improvement in cross-modal retrieval and 9.8% in unimodal settings. Despite its high retrieval accuracy, the model remains lightweight, requiring fewer parameters and yielding 8–10% F1-score gains on average, establishing a new state-of-the-art for scalable, sensor-agnostic RS-CMIR. 1
Establishing object correspondence between egocentric (ego) and exocentric (exo) views is a critical capability for robot learning and human-robot interaction. The core task involves segmenting an object in one view given a query mask from the opposing view. This is notoriously difficult due to cluttered scenes with many task-irrelevant objects and drastic appearance changes across perspectives. To address this, we introduce RegionAligner, a unified text-visual framework that strategically focuses learning on task-relevant regions. Our method first uses a large vision-language model to identify and name salient objects, effectively filtering out visual distractors. These object phrases are then fused with visual features from both views. We introduce a novel region-guided supervision strategy that promotes focus, enforces spatial alignment, and minimizes appearance disparity between the ego-exo views. Furthermore, our framework seamlessly adapts to unsupervised settings by automatically generating pseudo-labels from matched mask proposals, drastically reducing annotation costs. Extensive experiments on the challenging Ego-Exo4D dataset show RegionAligner significantly outperforms existing baselines, improving IoU by 10.16% (ego-to-exo) and 6.04% (exo-to-ego).
Work zones are essential to maintain, repair and upgrade our roadways. However, they introduce complex, dynamic and challenging environments for autonomous vehicles to navigate safely. To help address this challenge, we introduce the first publicly available, large-scale, multimodal work zone dataset collected with an autonomous vehicle consisting of multiple synchronized lidars and high-resolution cameras. Our WorkZone3D dataset covers various work zone elements such as cones, barrels, and channelizers, and provides 3D annotation boxes for these objects. We also propose a auto-annotation pipeline that can produce high-quality 3D labels, assimilating data across frames, even for rare classes which do not have pre-trained 3D object detection models to start with. We evaluate unimodal and multimodal models on our dataset, showing the critical role of sensor fusion in accurate 3D localization of such small objects at a distance, often having very few lidar points on them. Our results demonstrate the usefulness of WorkZone3D for generalization to real-world scenarios. Our code and dataset are available at https://github.com/ssuralcmu/WorkZone3D.git.
We present SOPHY, a generative model for 3D physics-aware shape synthesis. Unlike existing 3D generative models that focus solely on static geometry or 4D models that produce physics-agnostic animations, our method jointly synthesizes shape, texture, and material properties related to physics-grounded dynamics, making the generated objects ready for simulations and interactive, dynamic environments. To train our model, we introduce a dataset of 3D objects annotated with detailed physical material attributes, along with an efficient pipeline for material annotation. Our method enables applications such as text-driven generation of interactive, physics-aware 3D objects and single-image reconstruction of physically plausible shapes. Furthermore, our experiments show that jointly modeling shape and material properties enhances the realism and fidelity of the generated shapes, improving performance on both generative geometry and physical plausibility. Project page: https://xjay18.github.io/SOPHY.
Video question answering is a non-trivial task that demands joint understanding of visual contents and linguistic questions as well as temporal reasoning across video frames. Recent agent-based approaches address this by conducting multi-step reasoning with large language models (LLMs) across frame-level captions generated by vision-language models, but encounter limited temporal coherence across frames. A possible direction based on video language models (VideoLMs) directly captures temporal dynamics via video-level descriptions, but often lacks fine-grained visual cues due to a restricted number of input frames and a large dependency on input prompts. To tackle these challenges, we propose RefineVQA, a training-free framework that can easily be plugged into existing VideoLMs with iterative, LLM-guided description refinements. Specifically, the VideoLM produces an initial description, followed by LLM feedback determining whether the description suffices for the question and guiding further visual extraction, which in turn enhances the description quality while preserving temporal context. Plugged into state-of-the-art VideoLMs, ReFineVQA yields consistent gains across diverse benchmarks–NExT-QA, EgoSchema, VideoMME, ActivityNet, and StreamingBench–even with a small external LLM of 3.8B parameters.
The widespread use of synthetic image generation models and the challenges associated with authenticity preservation have fueled the demand for robust watermarking methods to safeguard authenticity and protect the copyright of synthetic images. Existing watermarking methods embed. Invisible signatures in synthetic images often compromise image quality and remain susceptible to multiple watermark removal attacks, including reconstruction and forgery methods. To overcome this issue, we propose a novel watermarking approach, AEON, which seamlessly integrates the watermark into the latent diffusion process and ensures the watermark aligns with scene semantics in the final image. Unlike existing invisible in-diffusion watermarking and traditional hash-based methods, our approach adapts the neural synthesized hash-based watermark to the semantics of the generated image during the intermediate diffusion process instead of embedding traditional hashes with the initial noise. Our proposed approach a) modulates the noise sampling in each diffusion denoising iteration through a learnable watermark embedding, b) optimizes consistency, reconstruction, and similarity loss, enforcing local and global alignment between the watermark structure and the underlying image content, and c) generates a strong watermark by allowing late embedding of the watermark in the diffusion process. Empirical results demonstrate the effectiveness of the proposed approach in retaining quality and its robustness against cumulative adversarial attacks.
Text-to-video generative models have emerged as powerful tools in content creation, capable of synthesizing highly realistic videos from textual prompts. However, the rapid advancement of these models introduces significant security and trust concerns - adversaries can now fabricate convincing videos, while existing detection methods struggle to generalize to unseen generative techniques. This is due to existing approaches’ reliance on supervised learning, requiring continuous dataset curation and retraining, which is impractical given the fast-paced evolution of generative models. In this work, we introduce the first training-free detection method for text-to-video generations, eliminating the need for labeled training data or prior exposure to generation techniques. Our approach exploits a fundamental weakness in text-to-video models: Unnatural temporal over-coherence in frame transitions. By leveraging a novel time-coherence detection criterion, our method identifies subtle inconsistencies in video embeddings, capturing temporal artifact patterns that reliably distinguish AI-generated content from real videos. We extensively evaluate our approach on an extended combination of benchmarks, amounting to 23 video generative models and 54.4K diverse videos, demonstrating that it outperforms the existing baselines, and shown to be effective on unseen generative models. This work establishes a new direction for training-free detection of text-to-video generated content, providing a scalable and time resilient solution.
Alzheimer’s disease (AD) is the most common form of dementia and a progressive, irreversible brain disorder that affects millions worldwide. The majority of existing research on AD classification relies on cross-sectional brain magnetic resonance imaging studies, which consider information from a single time point and fail to account for the progressive nature of AD. Longitudinal analysis, however, is crucial for capturing AD evolution and enabling more accurate diagnosis. To address this gap, we propose GeoHSAF, a novel hippocampus-based geometric learning framework for longitudinal AD classification. To overcome the challenge of missing or inconsistent hippocampal shapes across subjects and time points, our framework includes an interpolation module that predicts intermediate shapes, ensuring temporal continuity. We evaluate the effectiveness of GeoHSAF on three public longitudinal AD datasets: ADNI, OASIS, and AIBL, and benchmark its performance against existing approaches. GeoHSAF achieves new state-of-the-art results on binary classification tasks (AD vs. NC), while also demonstrating strong performance on more challenging triple-class classification tasks (AD vs. NC vs. MCI). Our work is fully reproducible, and all code is available at: https://github.com/ayodejimb/GeoHSAF
Event cameras are increasingly being explored for space applications due to their high dynamic range and increased spatiotemporal resolution. Existing datasets in this application have focused on capturing low-light, sub-pixel space objects and Earth observation scenarios. There remains a notable gap in datasets tailored to high-illumination conditions, particularly those involving direct solar imaging. This work introduces a dataset of solar event recordings captured with an event camera in a controlled sun-simulator environment. The dataset is specifically designed to support research in sun sensing and stray light analysis for spacecraft attitude estimation applications. It includes raw event data, annotated sun centroid locations, object motion profiles, and secondary optical aberration artifacts. In addition to the dataset, we present a systematic methodology for estimating the sun vector, intended to serve as a benchmark for evaluating sun sensing approaches in this application. All data and code are open source to facilitate further study: (https://github.com/sydneyid/SunE).
We present CURIO, an OCR system for low-resource historical manuscripts. In many challenging cases, manuscripts feature curved text lines, unsegmented lines with lack of spacing between words, and highly variable line lengths — conditions under which existing OCR methods fail. To tackle this challenge, we first extract lines and corresponding curvature profiles from manuscripts, then straighten them using a rectification procedure to reduce redundant background within each line. Because data is scarce, we compliment real data with synthetic data. To bridge the synthetic–real gap, we generate line images by warping rendered straight text along the rectified profiles, ensuring both real and synthetic lines align in their curvature characteristics. Our recognizer is a lightweight CNN–Transformer with padding-aware null activations, sparse attention and optimized with CTC loss for efficient training. We evaluate our method on challenging manuscript collections written in Sharada, a rare and endangered Indic script. CURIO outperforms strong CNN+RNN and Transformer baselines, with the largest gains on high-curvature lines and long lines. CURIO further transfers zero-shot to printed Sharada text, indicating robustness beyond manuscript domain.
Imperfect reconstructions arising from occlusions, shadows, reflections, and other factors during 3D scanning often result in incomplete sections of the scanned object, with missing parts scattered randomly across its surface. We introduce an uncertainty-aware signed distance field (SDF) latent transformer that leverages uncertainty to identify and reconstruct missing parts based on the global shape of the incomplete scanned object and the immediate neighborhood of the affected regions. To our knowledge, we are the first to utilize uncertainties for SDF shape completion in the latent space. Our model has been trained on the entire Objaverse 1.0 dataset and demonstrates that our uncertainty-aware SDF completion method significantly outperforms previous works both numerically and visually. Code will be published at github.com/cgtuebingen/ua3dscancomp.
Intrinsic image decomposition aims at separating an image into its underlying albedo and shading components, isolating the base color from lighting effects to enable downstream applications such as virtual relighting and scene editing. Despite the rise and success of learning-based approaches, intrinsic image decomposition from real-world images remains a significantly challenging task due to the scarcity of labeled ground-truth data. Most existing solutions rely on synthetic data as supervised setups, limiting their ability to generalize to real-world scenes. Self-supervised methods, on the other hand, often produce albedo-like maps that contain reflections and lack consistency under different lighting conditions. To address this, we propose SAIL, an approach designed to estimate illumination-invariant representations from single-view real-world images. We repurpose the prior knowledge of a latent diffusion model for unconditioned scene relighting as a surrogate objective for learning light-invariant estimates. To this end, we introduce a novel intrinsic image decomposition fully formulated in the latent space. To guide the training of our latent diffusion model, we introduce regularization terms that constrain both the lighting-dependent and -independent components of our latent image decomposition. Through our experiments, we demonstrate that SAIL produces stable albedo-like representations under varying lighting conditions and generalizes to multiple scenes, using only unlabeled multi-illumination data available online.
Synthetic data has played a pivotal role in developing large-scale 3D vision models due to its high-quality annotations and ease of curation. In domains where labeled data collection is difficult, such as endoscopy, synthetic data holds promise as a means to generate the large-scale annotated datasets required to train modern neural networks. In this work, we address a core question for data-scarce applications in 3D vision: how can we generate synthetic labeled data, and how useful would the data be for training downstream vision models? First, we introduce a novel data generation module that takes images with known geometry and camera poses as input and estimates the material and lighting conditions of the scene. To stabilize training, we leverage domain-specific properties like non-stationary lighting and anatomical material priors. We model the material properties as a bidirectional reflectance distribution function, parameterized by a neural network. Via the rendering equation, we can generate photorealistic images at arbitrary camera poses. We demonstrate that this method produces competitive novel view synthesis results compared to previous work while being more lightweight, flexible, and efficient. Second, we use our synthetic data to train models on various downstream 3D vision tasks and find that models trained solely on our synthetic data generally outperform those trained on real data across various metrics and tasks. Our experiments show that synthetic data is a promising avenue towards robust 3D vision in surgical scenes.1
Vision encoders power modern vision-only and vision-language systems, yet the geometry of their internal features remains opaque. In this work, we introduce a simple, general approach for vision latent analysis: reconstruct images from frozen encoder features and treat reconstructability as a proxy for retained information and feature organization. Concretely, we train a lightweight reconstructor to invert feature tensors and use it to compare various vision encoders — CLIP-based ViT, SigLIP, SAM, and InternViT. We rank models by the informativeness of their features and observe consistent gains with image-centric objectives and higher spatial resolution. Beyond measurement, controlled manipulations in feature space produce predictable pixel-level edits: orthogonal rotations (rather than spatial transformations) implement channel permutations and drive systematic color changes; linear contractions implement channel suppression; and a learned linear map enables plausible colorization of grayscale inputs. VLM-based experiments confirm that feature-space color swaps translate into semantic color changes in reconstructions. Our approach is encoder-agnostic in principle (demonstrated on ViT-based models), requires only access to features, and offers a practical diagnostic of what encoders remember, how that information is organized, and how it can be manipulated.
Identifying individual animals at different points in space and time is vital for effective wildlife monitoring and biodiversity conservation. While existing computer vision methods have shown promise in re-identifying animals, their capability in Animal Re-Identification (Animal ReID) remains restricted by the inherent visual variations, specifically high intra- and low inter-identity variations. High intra-identity variations refer to high visual diversity within the same individual due to pose or form changes and occlusions, and low inter-identity variations refer to subtle visual differences between distinct individuals due to fine-grained appearances. To address these challenges, we propose the Clip-based Animal RE-identification (CARE) framework, which leverages the image-conditioned textual description generation and individual-level semantic feature alignment, mitigating the negative impacts of visual variations in Animal ReID. Crucially, we have packaged CARE into a stand-alone toolkit and piloted it with stakeholders, facilitating real-world wildlife monitoring for biodiversity conservation. Extensive experiments on benchmark and in-the-wild datasets further demonstrate that CARE consistently outperforms state-of-the-art methods, validating its effectiveness in Animal ReID. Explore more about CARE at https://ml4sg.auckland.ac.nz/animal-re-identification-model/.
