Point cloud-based 3D single object tracking (3D SOT) plays a pivotal role in applications such as autonomous driving and robotic vision. Despite recent progress, most existing approaches rely solely on current-frame features for target localization. This approach overlooks temporal information that is crucial for robust tracking under occlusion, appearance variations, and sparse point clouds. In addition, the effectiveness of 3D SOT largely depends on the quality of feature fusion between the target template and the search region. Traditional fusion strategies often suffer from limited interaction capacity and weak discriminative representation. To address these challenges, we propose DT-Tracker, which performs multi-layer bidirectional feature interaction and temporal cue propagation to improve tracking robustness and feature discrimination capability. Specifically, we introduce a Dual-Context Propagation Network that applies bidirectional cross-attention across multiple layers between the template and search region, enabling deep semantic alignment and progressive feature refinement. Furthermore, we design a Temporal Context Fusion module that adaptively incorporates temporal cues from historical fusion features into the current frame, effectively improving resilience to occlusion and appearance drift. Extensive experiments on the KITTI and nuScenes datasets demonstrate that DT-Tracker achieves competitive results compared to existing representative methods.
The spatio-temporal relationship between the pixels of a video carries critical information for low-level 4D perception tasks. A single model that reasons about it should be able to solve several such tasks well. Yet, most state-of-theart methods rely on architectures specialized for the task at hand. We present L4P, a feedforward, general-purpose architecture that solves low-level 4D perception tasks in a unified framework. LAP leverages a pre-trained ViT-based video encoder and combines it with per-task heads that are lightweight and therefore do not require extensive training. Despite its general and feedforward formulation, our method is competitive with existing specialized methods on both dense tasks, such as depth or optical flow estimation, and sparse tasks, such as 2D/3D tracking. Moreover, it solves all tasks at once in a time comparable to that of single-task methods.
Efficiently reconstructing 3D scenes from monocular video remains a core challenge in computer vision, vital for applications in virtual reality, robotics, and scene understanding. Recently, frame-by-frame progressive reconstruction without camera poses is commonly adopted, incurring high computational overhead and compounding errors when scaling to longer videos. To overcome these issues, we introduce VideoLifter, a novel video-to-3D pipeline that leverages a local-to-global strategy on a fragment basis, achieving both extreme efficiency and SOTA quality. Locally, VideoLifter leverages learnable 3D priors to register fragments, extracting essential information for subsequent 3D Gaussian initialization with enforced inter-fragment consistency and optimized efficiency. Globally, it employs a tree-based hierarchical merging method with key frame guidance for inter-fragment alignment, pairwise merging with Gaussian point pruning, and subsequent joint optimization to ensure global consistency while efficiently mitigating cumulative errors. This approach significantly accelerates the reconstruction process, reducing training time by over 82 % while achieving better visual quality than SOTA methods.
High-quality, real-time dynamic scene reconstruction and rendering are essential for immersive applications. While techniques like 3D Gaussian Splatting (3DGS) succeed in static scenes, dynamic monocular scenarios still suffer from deformation, surface noise, and inconsistent view-dependent effects due to insufficient geometric constraints and inadequate multi-view consistency enforcement for monocular input. To address these challenges, we present Normal-Regularized and Multi-View-Consistent Gaussian Splatting (NVC-GS), a novel approach combining geometry-aware normal regularization with diffusion-based multi-view consistency. Our method explicitly preserves geometric consistency in dynamic objects through tailored normal constraints, while leveraging diffusion-driven latent space regularization to ensure cross-view rendering consistency, particularly for complex materials such as reflective surfaces. Experimental results demonstrate that our approach effectively improves geometric accuracy and visual quality in dynamic scenes while maintaining real-time capabilities, outperforming existing methods in terms of deformation handling, surface noise reduction, and rendering of reflective materials.
Self-supervised learning has emerged as a promising approach for acquiring transferable 3D representations from unlabeled 3D point clouds. Unlike 2D images, which are widely accessible, acquiring 3D assets requires specialized expertise or professional 3D scanning equipment, making it difficult to scale and raising copyright concerns. To address these challenges, we propose learning 3D representations from procedural 3D programs that automatically generate 3D shapes using simple 3D primitives and augmentations. Remarkably, despite lacking semantic content, the 3D representations learned from the procedurally generated 3D shapes perform on par with state-of-the-art representations learned from semantically recognizable 3D models (e.g., airplanes) across various downstream 3D tasks, such as shape classification, part segmentation, masked point cloud completion, and both scene semantic and instance segmentation. We provide a detailed analysis on factors that make a good 3D procedural programs. Extensive experiments further suggest that current 3D self-supervised learning methods on point clouds do not rely on semantics of 3D shapes, shedding light on the nature of 3D representations learned.
Decomposing scene radiance into physically meaningful components, including direct reflection, interreflection, and scattering, enables a deeper understanding of scene appearance. In this paper, we propose the first method to perform multi-range radiance component separation using only events captured by an event camera, without requiring any additional frame-based measurements. Our approach scans the scene by swiping line-shaped illumination across it, while exploiting the event camera's high temporal resolution and wide dynamic range to recover both direct and multiple global components corresponding to different light propagation distances. To address the noise inherent in event-integration-based radiance recovery, we present a pixel-wise calibration strategy that leverages the reproducibility of per-pixel noise patterns. We demonstrate that this calibration is highly effective in suppressing noise, enabling stable recovery from subtle signals. Moreover, we show that by detecting the timing at which the scanning line passes each pixel, the same line-scan event data can be exploited for coarse 3D reconstruction. Experimental results on real scenes show that our event-based approach achieves faster and finer component separation, while also enabling coarse depth estimation without the exposure control required by frame-based cameras.
Stereo matching networks can suffer from generalization challenges when trained on synthetic data and deployed in real-world settings. While existing methods rely on fine-tuning or pre-trained vision foundation models for cross-domain robustness, we revisit this gap from a training perspective and explore a structure-grounded training design that directly improves generalization of RNN-based stereo matching models using only a limited amount of synthetic stereo data, without changing the network architecture or adding any inference overhead. Specifically, we target all three main modules of a typical stereo matching pipeline: in cost volume construction, we enhance geometric cues through data augmentation; in context encoding, we strengthen semantic guidance via auxiliary multitask context supervision; in recurrent disparity refinement, we regulate update dynamics with depth-update regularization. Experiments on multiple mainstream architectures and diverse real-world datasets suggest consistent gains in robustness, improving RAFT-Stereo by 6.6% on KITTI 2015, IGEV-Stereo by 13.7 % on Middlebury, and DLNR by 55.4 % on ETH3D. These insights reveal the previously overlooked importance of structure-grounded training design for achieving reliable stereo depth estimation under data-scarce, domain-shifted conditions.
Point-based representations have recently gained popularity in neural rendering. While they offer many advantages, rendering them from close-up views often results in holes. In splatting-based neural point renderers, these are caused by gaps between different splats, which cause many rays to not intersect with any splat when viewed close-up. A different line of work uses attention to estimate each ray's intersection by interpolating between nearby points. Our work builds on one such method, known as Proximity Attention Point Rendering (PAPR), which learns parsimonious and geometrically accurate point representations. While in principle PAPR can fill holes by learning to interpolate between nearby points appropriately, PAPR also produces holes when rendering close-up, as the intersection point is often predicted incorrectly. We analyze this phenomenon and propose two novel solutions: a method for dynamically selecting nearby points to a ray for interpolation, and a robust attention method that better generalizes to local point configuration around unseen rays. These significantly reduce the prevalence of holes and other artifacts in close-up rendering compared to recent neural point renderers.
Monocular 3D object detection aims to predict object category, position, size, and orientation from a single RGB image. Existing DETR-based monocular 3D detectors suffer from maintaining consistent high-confidence responses due to weak or incomplete target features, resulting in information loss for distant and occluded objects during encoding-decoding. Firstly, to address the core challenge of insufficient target feature perception in complex scenes, we propose a staged iterative monocular 3D detector that progressively refines targets from coarse to fine through multiple paired encoding-decoding stages, significantly improving both feature utilization and network convergence. Furthermore, each stage integrates a dynamic target iteration module that continuously enhances query representation by focusing on high-confidence regional features, thereby enhancing the model's perception of potential targets. Finally, we design a dual-branch depth estimator with parallel global and local processing for a comprehensive representation of the depth feature. Experimental results show that our method achieves superior performance over prior approaches on challenging scenarios (e.g., distant and occluded objects) in the KITTI dataset without auxiliary data, while maintaining competitive accuracy on the nuScenes benchmark under frontal-view settings.
We introduce a novel dataset and evaluation approach for long-range depth prediction of small objects that enables consistent comparison across direct time-of-flight (ToF) sensors and learned depth estimation methods. In autonomous driving, accurate depth perception is essential for identifying and locating surrounding elements and determining safe driving paths. Traditional depth metrics focus on distance accuracy but fail to evaluate a key factor at long ranges: distinguishing small, slightly elevated structures from the ground - crucial for anticipating obstacles and making safe driving decisions. At far distances, imagebased systems suffer from resolution limitations that tend to oversmooth the ground plane, causing elevated objects to be mistaken as texture patterns on the surface. Conversely, scanning LiDAR systems may return only a single point from an elevated object due to steep incident angles and sparse returns, preventing accurate differentiation from the ground. This hampers a fair comparison of object presence and shape. To address this, we propose a framework that evaluates how well the estimated point clouds preserve semantic content relative to ground-truth data. We leverage neural network-based feature extraction to assess structural similarity, enabling a modality-agnostic evaluation of object-level fidelity. Our method also supports analysis of the trade-off between resolution and accuracy, investigating performances across sensor types - such as highresolution cameras versus LiDAR - and conditions, including day and night scenarios. This enables a more comprehensive understanding of the capabilities and limitations of current depth prediction approaches in real-world settings.
Reconstructing complete 3D scenes from extremely sparse viewpoints (e.g., 2-3 wide-baseline images) remains a core yet unsolved challenge. Existing 3D Gaussian Splatting (3DGS) and neural rendering methods degrade severely when view overlap is limited, often producing incomplete or geometrically distorted results. We introduce SplatFusion, a reconstruction framework that requires no training or fine-tuning of diffusion models, instead relying solely on pretrained video diffusion priors to synthesize missing scene content plausibly. Our core idea is a Scene-Consistent Temporal Guidance (SCTG) mechanism that tightly couples 3D structure with generative diffusion models. Specifically, SCTG conditions video diffusion on sequences rendered from the evolving 3DGS representation, enforcing both spatial alignment with geometry and temporal coherence across synthesized frames. These refined views are back-projected to densify and correct the 3D scene iteratively. Extensive experiments on diverse realworld datasets demonstrate that SplatFusion consistently outperforms existing sparse-view reconstruction methods. Evaluations using VLM-based perceptual scores and the MEt3R metric for geometric consistency show clear gains in visual fidelity and temporal coherence, even in scenarios where previous approaches fail. Our training-free framework opens new possibilities for practical 3D reconstruction applications where dense view acquisition is impractical.
3D edge reconstruction from posed multi-view images remains a critical yet under-explored task. While 3D Gaussian Splatting (3DGS)-based methods have recently achieved promising performance, they face two main challenges. First, edges exhibit clear discontinuities from the background, but the intrinsic smoothness of Gaussian kernels makes it challenging to model such discontinuities. Second, due to the absence of multi-view edge annotations, models are trained with pseudo labels instead. These pseudo labels extracted by pre-trained $2 D$ edge detectors often exhibit cross-view inconsistencies, leading to degraded performance. To address these issues, we propose a novel uncertainty-aware 3D edge reconstruction using Difference of Gaussians (DoG) as kernels, called EdgeDoG. First, we incorporate DoG kernels to model edge discontinuities explicitly. Second, we design a dual-uncertainty strategy: primitive-level uncertainty is estimated via multi-view Fisher information to eliminate noisy 3D primitives, while pixel-level uncertainty is computed from gradients of rendered depth maps to reweight the training loss, thereby compensating for inconsistent 2D pseudo labels with robust 3D geometric cues. Extensive experiments on diverse datasets demonstrate that our method achieves superior performance compared to previous approaches.
In sparse novel view synthesis with few input views and wide baselines, existing methods often fail due to weak geometric correspondences and view-dependent color inconsistencies. Splatting-based approaches can produce plausible results near training views, but they frequently overfit and struggle to maintain smooth, realistic appearance transitions in novel viewpoints. We introduce a splat-based gradient-domain fusion method that addresses these limitations. Our approach first establishes reliable dense geometry via two-view stereo for stable initialization. We then generate intermediate virtual views by reprojecting input images, which provide reference gradient fields for gradient-domain fusion. By blending these gradients, our method transfers low-frequency, view-dependent colors to the rendered Gaussians, producing seamless appearance transitions across views. Extensive experiments show that our approach consistently outperforms state-of-the-art sparse Gaussian splatting methods, delivering robust and perceptually plausible view synthesis. A comprehensive user study further confirms that our results are perceptually preferred, with significantly smoother and more realistic color transitions than existing methods.
Understanding how humans interact with objects is key to building robust human-centric artificial intelligence. However, this area remains relatively unexplored due to the lack of large-scale datasets. Recent datasets focusing on this issue mainly consist of activities captured entirely in controlled lab environments, and contact annotations are mostly estimated using threshold clips. We introduce Contact4D, a multi-view video dataset for humanobject interaction that provides detailed body poses and accurate contact annotations. We use a flexible multiview capture system to record individuals performing furniture assembly tasks and provide annotations for human detection, tracking, 2D/3D pose estimation, and groundtruth contact. Additionally, we propose a novel processing pipeline to extract accurate hand poses even when they are severely occluded. Contact4D consists of $2 M$ images captured from 19 synchronized cameras across 350 video sequences, spanning diverse environments, varioius furniture types, and unique subjects. We evaluate existing methods for human pose estimation and human-centric contact estimation, demonstrating their inability to generalize to our dataset. Lastly, we fine-tune a pretrained MultiHMR model on Contact4D and observe an improved performance of 56.6% body MPJPE and 26.4% hand MPJPE in scenarios under severe self-occlusion and object occlusion. Code and data are available at https://jyuntins.github.io/Contact4D.
The development of multi-view image synthesis is constrained by the scarcity of training data. One promising solution is to finetune well-trained video generative models to synthesize 360-degree videos of objects. While these methods benefit from the strong generative priors inherited from the pretrained knowledge, they are limited by the high computational costs incurred by the large number of viewpoints. Existing methods commonly adopt temporal attention mechanism to address this. However, these methods suffer from undesirable artifacts such as 3D inconsistency and over-smoothing in the generated results. In this paper, we introduce a novel approach to unlock the video priors for multi-view synthesis by reducing generation into a sparser yet more precise process. Specifically, we introduce two strategies to achieve this: i) Condensing the video diffusion model to synthesize highly consistent sparse multiview images. ii) Extracting dense geometrical priors from the pretrained video diffusion models to enhance the generation stability. The combination of these two strategies formulates a novel framework for multi-view synthesis, which is capable of synthesizing highly consistent sparse multiview images with strong generalization ability. Extensive experiments demonstrate that our approach achieves superior efficiency, generalization, and consistency, outperforming state-of-the-art multi-view synthesis methods.
In this paper, we explore the potential of snapshot compressive imaging (SCI) for dynamic 3D scene reconstruction from a single temporal compressed image. SCI is a low-cost imaging technique that captures high-dimensional information-such as temporal data-using $2 D$ sensors and coded masks, significantly reducing data bandwidth while offering inherent privacy advantages. While recent advances in Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have enabled 3D reconstruction from SCI measurements, these methods are fundamentally limited to static scenes and fail to generalize to dynamic content. To address this, we propose SCIGaussian-D, a novel framework that enables dynamic 3D reconstruction from a single SCI image. Our method represents the scene with 3D Gaussians defined in a canonical space and models motion using learnable deformation fields. By incorporating the SCI imaging model into the training loop, SCIGaussian-D directly reconstructs the dynamic 3D scene and recovers the corresponding camera motion from a single SCI. We evaluate our method on both synthetic and real SCI datasets, demonstrating significant improvements in reconstruction quality over existing baselines. Our results establish a new state of the art for dynamic scene reconstruction within the SCI framework, paving the way for practical applications in high-speed imaging and real-time scene rendering.
Creating interior layout designs has numerous applications, including virtual reality, architectural visualization and real estate planning. Generating realistic and functional indoor scenes requires a nuanced understanding of spatial configurations and human-centered design principles. We propose InteriorAgent, an LLM-agent-driven framework for text-to-3D indoor scene generation that produces scenes with visual quality and functional utility that significantly surpass prior works. We achieve this through several key advantages of InteriorAgent: (1) encoding of interior design principles with a novel scene description language, (2) aesthetics and functionality through synthesis tools that satisfy design principles, (3) realism and prompt adherence with optimization tools that ensure ergonomics and iterative constraint satisfaction, (4) extensibility with a framework that allows incorporating even mature, complex tools like diffusion models, LLMs and 3D generation repositories. We evaluate InteriorAgent through a user study, where participants strongly favor its generated scenes over prior state-of-the-art methods. Additionally, we demonstrate novel applications uniquely enabled by InteriorAgent, including language-based scene editing and seamless tool integration for new tasks. Code and data will be publicly released.
Fast and accurate 3D shape generation from point clouds is essential for real-world applications such as robotics, AR/VR, and digital content creation. We present ConTiCoM-3D, a continuous-time consistency model that generates 3D shapes directly in point space, without relying on discretized diffusion steps, pre-trained teacher models, or latent-space encodings. Our approach combines a TrigFlow-inspired continuous noise schedule with a Chamfer Distance-based geometric loss, providing stable training in high-dimensional point sets while avoiding costly Jacobian-vector products. This enables efficient one- to two-step inference with high geometric fidelity. Unlike previous methods that require iterative denoising or latent decoders, ConTiCoM-3D operates entirely in continuous time with a time-conditioned neural network, achieving fast generation. Extensive experiments on the ShapeNet benchmark demonstrate that our method matches or surpasses leading diffusion and latent consistency models in both quality and efficiency, establishing ConTiCoM-3D as a practical solution for scalable 3D shape generation.
Understanding dynamic 3D scenes is crucial for extended reality (XR) and autonomous driving. Incorporating semantic information into 3D reconstruction enables holistic scene representations, unlocking immersive and interactive applications. To this end, we introduce TRASE, a novel tracking-free 4D segmentation method for dynamic scene understanding. TRASE learns a 4D segmentation feature field in a weakly-supervised manner, leveraging a soft-mined contrastive learning objective guided by SAM masks. The resulting feature space is semantically coherent and well-separated, and final object-level segmentation is obtained via unsupervised clustering. This enables fast editing, such as object removal, composition, and style transfer, by directly manipulating the scene's Gaussians. We evaluate TRASE on five dynamic benchmarks, demonstrating state-of-the-art segmentation performance from unseen viewpoints and its effectiveness across various interactive editing tasks. Our project page is available at: https://yunjinli.github.io/project-sadg/
In human-object interaction (HOI), physical contact between the body and objects is a primary determinant of realism and plausibility. Prior HOI methods typically encode relations via global joint-to-centroid or joint-to-boundary. Such strategies neglect contact anchors that are essential for defining joint-to-contact relations-where and how HOI occurs, thereby implicitly reducing the problem to nearest-distance optimization. Without explicit contact anchors and joint-to-contact dynamics, previous models drift toward artifacts: human-object penetration or unnatural object floating. We argue that modeling contact relationships by contact anchors is important for generating realistic HOIs, as it directly captures where and how humans physically interact with objects rather than merely minimizing spatial proximity. To address these limitations, we propose Contact-Aware HOI (ContA-HOI), a progressive framework that decomposes HOI generation into three synergistic stages: discovering where contact occurs, modeling how contact evolves, and guiding generation with contact constraints. First, a Contact Affordance Predictor (CAP) addresses the “where” by predicting precise object-surface contact anchors from text, human pose, and object geometry. Second, these anchors seed a Contact Relation Field (CRF) that captures “how” by modeling spatiotemporal dynamics of joint-to-contact relations throughout the interaction. Finally, a Contact Dynamics Model (CDM) learns a prior CRF evolution pattern and guides motion diffusion sampling by aligning the generated motion's CRF with this learned prior. On the FullBodyManipulation dataset, ContA-HOI yields more realistic and physically plausible HOIs, improving foot sliding and contact percentage over recent baselines.
