Recent advances in transformer-based text-to-motion generation have significantly improved motion quality. However, achieving both real-time performance and long-horizon scalability remains an open challenge. In this paper, we present Hi-RQCT (Hierarchical Residual-Quantized Causal Transformer), which generates high-quality lifelike 3D human motions by training a single transformer model. Hi-RQCT consists of only two main components: 1) RVQ-VAE, a hierarchical residual vector quantization variational autoencoder, which discretizes continuous motion sequences with high precision; 2) Hierarchical Causal Transformer, responsible for generating the base motion sequences in an autoregressive manner while simultaneously inferring residuals across different layers. Experimental results demonstrate that Hi-RQCT can generate smooth and continuous motion sequences up to 260 frames (13 seconds), surpassing the 196 frames (10 seconds) length limitation of existing datasets like HumanML3D. On the HumanML3D test set, our model achieves the best quantitative performance, and the generated motions also exhibit highly realistic and expressive visual quality in qualitative evaluations.
Consistent colour appearance across shots is essential in motion picture post-production, particularly during the colour-matching stage that ensures scene coherence after initial white balancing. While there is a large body of research targeting illuminant estimation for single images, achieving consistent relative colour appearance across multiple shots (colour matching) remains an underexplored challenge. To address this, we propose SqueezeTransformerNet, a novel hybrid neural architecture designed for multi-shot scene colour consistency. It processes selected keyframes using a SqueezeNet-based CNN (from FC4) for efficient local feature extraction and employs a Transformer encoder to explicitly model inter-shot dependencies, yielding temporally coherent relative illuminant estimations across a scene. Temporal illuminant consistency between frames is measured by introducing a new metric: the Mutual MSE, which we target as part of our losses. For evaluation, we curated multi-shot scene benchmarks by repurposing the standard Gehler-Shi and NUS8 datasets. On these custom scene-level benchmarks and actual TV series production data (112 scenes and 1380 shots), SqueezeTransformerNet shows superior performance over C-4 in MSE (x8.8-x15.6) and Mutual MSE (x1.2-x2.3) consistency metrics. SqueezeTransformerNet also achieves better Angular Error (AE) than the FC4 network on single-frame colour constancy benchmarks. Trained across diverse data, the network exhibits robustness to different cameras, offering a promising solution for automated, temporally consistent colour matching in post-production workflows.
Real-world materials, such as those desirable to render accurately in production, exhibit complex appearances that vary spatially across their surfaces. There are many works for estimating material-scattering properties, both for analytic and data-driven models, but common limitations of recent effort include the use of synthesized rather than measured data, reliance on neural networks that do not easily fit rendering pipelines, and not accounting for the spatially varying nature of light-scattering response over real surfaces. We propose a pipeline for predicting, measuring, and rendering spatially varying appearances using a custom variable-illumination sphere with concepts that may be applied to other systems such as goniophotometers. We build on the adaptive-parameterization technique of Dupuy and Jakob, extending the ideas to support spatially varying material measurement. Focus is given to practical, efficient capture of varied scattering responses over a sample surface. Metrics and clustering techniques using data from an abbreviated initial capture are used to improve the efficiency of the full acquisition and material representation. Our work introduces a novel approach to capture complex, spatially varying appearances more efficiently and also provides data for material and pattern analysis that may be used for rendering or other applications.
The latent space of Generative Adversarial Networks (GANs) forms a continuous manifold where each point corresponds to a realistic image. Traversing linear paths within this space yields smooth transitions in image appearance, preserving realism without abrupt changes. For example, interpolating between latent codes of a closed-mouth and an open-mouth subject produces a natural animation of mouth movement. Building on this property, we propose 3D-LaSR, a novel method for reenacting 3D human heads using latent space paths, derived from a single monocular video. Our approach leverages GAN inversion to extract latent vectors from video frames, and supports expression transfer between identities by applying latent animation paths from one subject to another. Consequently, our method outputs dynamic sequences of latent vectors. Unlike decoder-based techniques that produce fixed geometry, 3D-LaSR can directly manipulate the scenes using established GAN editing techniques to modify attributes such as hairstyle, age, eyewear, or expression. The method uses a key frame inversion approach, where intermediate frames are synthesized through linear interpolation. This significantly reduces computational cost compared to methods that require per-frame inversion. We validate our approach through a comprehensive set of experiments, demonstrating high-quality video synthesis, effective expression transfer, and flexible editability. Finally, as we develop our method around state-of-the-art 3D Gaussian splatting GANs, our method is able to render in explicit 3D environments such as video engines or VR settings.
We present a novel framework for scene representation and rendering that combines 3D Gaussian splatting with dual spherical harmonics decomposition to enable high-quality novel view synthesis under arbitrary lighting conditions. Our key innovation lies in separately modeling diffuse and specular reflection components using distinct directional parameterizations: view direction for diffuse components and reflection direction for specular components. This dual spherical harmonic (SH) decomposition, enables realistic rendering of reflective surfaces while maintaining the computational efficiency of Gaussian splatting. To address geometric fidelity challenges, our method incorporates a comprehensive optimization strategy including silhouette-guided geometric constraints for sharp opacity boundaries, HDR saturation loss for handling bright regions in high dynamic range environments, and multi-faceted normal vector optimization with photometric and physics-based supervision. Experimental validation on the DiLiGenT-MV benchmark demonstrates superior performance compared to existing methods, with significant improvements in PSNR, SSIM, and LPIPS metrics while achieving fast optimisation and real-time rendering performance suitable for interactive applications.
