Human motion data is inherently rich and complex, containing both semantic content and subtle stylistic features that are challenging to model. We propose a novel method for effective disentanglement of the style and content in human motion data to facilitate style transfer. Our approach is guided by the insight that content corresponds to coarse motion attributes while style captures the finer, expressive details. To model this hierarchy, we employ Residual Vector Quantized Variational Autoencoders (RVQ-VAEs) to learn a coarse-to-fine representation of motion. We further enhance the disentanglement by integrating codebook learning with contrastive learning and a novel information leakage loss to organize the content and the style across different codebooks. We harness this disentangled representation using our simple and effective inference-time technique Quantized Code Swapping, which enables motion style transfer without requiring any fine-tuning for unseen styles. Our framework demonstrates strong versatility across multiple inference applications, including style transfer, style removal, and motion blending.
Sketching is a direct and inexpensive means of visual expression. Though image-based sketching has been well studied, video-based sketch animation generation is still very challenging due to the temporal coherence requirement. In this paper, we propose a novel end-to-end automatic generation approach for vector sketch animation. To solve the flickering issue, we introduce a Differentiable Motion Trajectory (DMT) representation that describes the frame-wise movement of stroke control points using differentiable polynomial-based trajectories. DMT enables global semantic gradient propagation across multiple frames, significantly improving the semantic consistency and temporal coherence, and producing high-framerate output. DMT employs a Bernstein basis to balance the sensitivity of polynomial parameters, thus achieving more stable optimization. Instead of implicit fields, we introduce sparse track points for explicit spatial modeling, which improves efficiency and supports long-duration video processing. Evaluations on DAVIS and LVOS datasets demonstrate the superiority of our approach over SOTA methods. Cross-domain validation on 3D models and text-to-video data confirms the robustness and compatibility of our approach.
Diffusion-based scene text synthesis has progressed rapidly, yet existing methods commonly rely on additional visual conditioning modules and require large-scale annotated data to support multilingual generation. In this work, we revisit the necessity of complex auxiliary modules and further explore an approach that simultaneously ensures glyph accuracy and achieves high-fidelity scene integration, by leveraging diffusion models' inherent capabilities for contextual reasoning. To this end, we introduce TextFlux, a DiT-based framework that enables multilingual scene text synthesis. The advantages of TextFlux can be summarized as follows: (1) OCR-free model architecture. TextFlux eliminates the need for OCR encoders that are specifically used to extract visual text-related features. (2) Strong multilingual scalability. TextFlux is effective in low-resource multilingual settings, and achieves strong performance in newly added languages with fewer than 1,000 samples. (3) Streamlined training setup. TextFlux is trained with only 1% of the training data required by competing methods. (4) Controllable multi-line text generation. TextFlux offers flexible multi-line synthesis with precise line-level control, outperforming methods restricted to single-line or rigid layouts. Extensive experiments and visualizations demonstrate that TextFlux outperforms previous methods in both qualitative and quantitative evaluations. Our code is available at https://github.com/yyyyyxie/textflux.
Accurately measuring real-world object dimensions from Remote Sensing (RS) images is crucial for applications in geospatial analysis and urban planning. Traditional Vision-Language Models (VLMs) struggle with spatial reasoning, while end-to-end remote sensing VLMs are often limited to predefined tasks such as image captioning. In this paper, we propose a visual agentic system for spatial metric query answering, dynamically integrating code-generation agents with a grounded remote sensing VLM and a Vision Specialist. Our system autonomously identifies reference objects, infers scale factors, and performs spatial measurements through structured subroutines. Experiments demonstrate that our approach achieves higher accuracy in footprint area estimation compared to state-of-the-art large language models with vision capabilities.
Correctly matching real-world environment lighting conditions is an important step in making Augmented Reality content better fit with surrounding real objects. It is also the first step in larger, more complex problems like object relighting, shadow estimation, surface shading, etc. Dynamic classification of lighting conditions thus needs to be robust and lightweight. In this paper, we investigate the suitability of using pure EXIF data for classifying outdoor lighting conditions in four broad categories using a variety of shallow machine learning models. We gather a dataset of images together with EXIF metadata to test different models and show the results from the best-performing one in a real-time Augmented Reality application on a smartphone.
In recent years, radiance field methods, and in particular 3D Gaussian Splatting (3DGS), have distinguished themselves in the field of image-based rendering and scene reconstruction techniques, gaining significant success in academia and being cited in numerous research papers. Like other methods, 3DGS requires a large and diverse dataset of images for network training as a fundamental step to ensure effectiveness and high-quality results. Consequently, the acquisition phase is highly time-consuming, especially considering that a portion of the acquired dataset is not actually used for training but is reserved for testing. This is necessary because all commonly used metrics for evaluating the quality of 3D reconstructions, such as PSNR and SSIM, are reference-based metrics; i.e., requiring a ground truth. In this work, we present NOVA, a study focused on no-reference evaluation of 3DGS renders, based on key metrics in this field: PSNR and SSIM.
In computer graphics, mesh clustering is a key component of various applications such as shape matching or skinning weight computation, especially when using hierarchical clustering. Garland et al. [GWH01] proposed to build a hierarchy of clusters by simplifying the dual graph of the mesh. We extend their method to provide control over cluster shapes through a combination of error metrics. Additionally, we alleviate the challenging task of finding an optimal threshold (stopping criterion) by considering a weighted feature graph that incorporates persistent cluster information throughout the hierarchy.
In this work, we present a gaze prediction model for Virtual Reality task-oriented environments. Unlike past work which focuses on gaze prediction for specific tasks, we investigate the role and potential of temporal continuity in enabling accurate predictions in diverse task categories. The model reduces input complexity while maintaining high prediction accuracy. Evaluated on the OpenNEEDS dataset, it significantly outperforms baseline methods. The model demonstrates strong potential for integration into gaze-based VR interactions and foveated rendering pipelines. Future work will focus on runtime optimization and expanding evaluation across diverse VR scenarios.
In this paper, we propose a method to improve the performance of hand pose estimation from egocentric view. To accurately capture hands moving within a wide range in daily activities, we mounted a fisheye stereo camera on a head mounted display to obtain wide-angle images from egocentric view. Our proposed two-stage method addresses the camera distortion introduced by this setup. The 2D hand keypoints estimated by stage-1 HandNet are converted into 3D hand keypoints through triangulation for perspective cropping. Stage-2 HandNet then predicts the final 2D hand keypoints from the undistorted hand crop image. To train stage-1 HandNet for perspective cropping, we built FisheyeEgoHAND dataset which consists of three categories of scenarios (separate hand, hand-hand, and hand-object) that reflect various hand interactions in an egocentric view. Through experiments, we demonstrated that two-stage 2D hand pose estimation outperforms one-stage approach without perspective cropping.
The VisibleUS project aims to generate synthetic ultrasound images from cryosection images, focusing on the musculoskeletal system. Cryosection images provide a highly accurate representation of real tissue structures without artifacts. Using this rich anatomical data, we developed a ray-tracing-based simulation algorithm that models ultrasound wave propagation, scattering, and attenuation. This results in highly realistic ultrasound images that accurately depict fine anatomical details, such as muscle fibers and connective tissues. The simulation tool has various applications, including generating datasets for training neural networks and developing interactive training tools for ultrasound specialists. Its ability to produce realistic ultrasound images in real time enhances medical education and research, improving both the understanding and interpretation of ultrasound imaging.
Successful scene arrangement requires ensuring appropriate distances between objects and avoiding excessive overlaps or separations. This work proposes a method for automatically learning spatial relationships between objects in scene arrangement using a differentiable renderer loss. First, objects surrounding a dominant item (e.g., a table in a dining room) are identified and represented as nodes in a polygon that encodes their spatial relations. The difference between the predicted and ground truth polygons is minimized via a rendering loss, which is integrated into the training of a generative diffusion model. This approach continuously optimizes the spatial distribution of objects during generation, ensuring physical consistency and practical usability. Experimental results show a significant reduction in collision rates compared to state-of-the-art methods.
We investigate the problem of robust and real-time rendering of algebraic surfaces. We show that expressing the intersection of the ray and the algebraic surface as a single univariate polynomial is not robust in practice, comparing results between monomial, Bernstein, Lagrange, and Chebyshev basis fits. We show that fitting multiple polynomials over subintervals, such as a unit length subdivision of the ray extent within the region of interest, improves robustness at a negligible performance cost.
Hair braids are widely used in various games and animated movies, thanks to their simplified representation and ease of animation. However, the existing research on modeling braids often relies on a limited dictionary of commonly seen hair braid patterns, constraining artists' ability to experiment by creating imaginary or creative hair braids. In this paper, we introduce a simple sketch-based interface for creating arbitrary hair braids. Our method employs a two-stage framework that first interprets a user-drawn sketch to extract the braid pattern. To accommodate arbitrarily drawn sketches, we then use a physics-inspired simulation to generate visually pleasing braids. In addition to automatically generating braids, our system allows users to interactively refine the braid pattern to create braids that match the user's imagination, facilitating experimentation and exploration of different braid structures.
Implicit geometry is a popular representation for shape modelling. It provides several interesting properties, such as infinite resolution, continuity and smooth blending. However, implicit surfaces are difficult to deform as deformations need to be invertible. They are in general restricted to linear representations or more advanced translation-based deformations. We propose a method that adapts cage-based deformation to implicit surfaces while handling self-intersections in the deformed space.
We present an automated framework for counting and measuring the polyps of Cladocora caespitosa, a Mediterranean reef-building coral. To our knowledge, the most practical method for counting polyps currently involves ecologists' visual inspection of a 3D model. However, measuring polyps from the model can lead to inaccuracies due to distortions in the reconstruction. Our method integrates deep learning-based instance segmentation on 2D images with 3D models for unique polyp identification, ensuring precise biometric extraction. The proposed pipeline automates polyp detection, counting, and measurement while overcoming the limitations of manual in situ methods. Laboratory validation demonstrates its accuracy and efficiency, paving the way for scalable, high-resolution phenotyping, and field monitoring of Mediterranean coral populations.
