Human motion synthesis has advanced significantly in domains such as text-to-motion and gesture generation, yet synthesising realistic interaction motion between multiple characters remains relatively underexplored, particularly in dynamic and user-driven environments like virtual reality (VR). Existing approaches rely on constructing sequences across a set number of frames, inherently limiting their usability in real-time and open-ended applications such as virtual reality. In this work, we present a novel latent diffusion-based framework for real-time interaction motion synthesis. Unlike prior models that operate on fixed-length sequences, our system continuously adapts to user behaviour, generating contextually appropriate and temporally coherent response motions. We introduce Interaction Contact Labels, a generalisation of ground contact annotations, to capture physical interdependencies between interacting characters and integrate them into the training objective to improve realism. Our approach enables open-ended, responsive character interactions suitable for live applications in VR, AR, and robotics. Further to this, we demonstrate a real-time implementation of the proposed method, which utilises CycleGANs to combine webcam data with high-quality motion capture data, allowing for easy end-user set-up.
Smearing is an essential effect to expressively convey motion in stylized animations. In this paper, we extend the method of Basset et al. [2024] to better emphasize the main motion's trajectory of an object when generating elongated in-betweens, i.e., when stretching a 3D object along its trajectory to cover adjacent frames. This limits visual artifacts such as intersections that typically occur when trajectories self-overlap due to local rotations or abrupt changes of direction (trajectories with high curvatures or even discontinuities at contacts). We address these cases with minor computational and memory overheads, and offer enhanced impact expressiveness by combining smear and squash-and-stretch effects at collisions.
Recent technological innovations, especially in extended reality (XR) and artificial intelligence (AI), redefine storytelling approaches. These innovations are expanding creative possibilities for filmmakers while transforming how audiences engage with and experience films and other entertainment products. Despite technological advancements, the pre-production phase depends primarily on traditional planning and visualization methods. This research proposes a novel paradigm to create storyboards in XR, leveraging the capabilities of object-detection and pose-estimation systems to benefit the storyboarding phase. An application has been designed and developed to create 3D storyboards on a real scale within a physical environment and all its furniture. Wearing a head-mounted display for XR, users can move in the physical space, enrich it with virtual elements and characters, and frame the environment to obtain storyboard panels that mix real and virtual elements. The proposed system has been tested to assess its usability, and initial findings indicate that users have appreciated this application.
Haptic technology has the potential to substantially support users in mastering complex activities and improving their performance. This paper presents a haptic assistance system for targeting in virtual reality (VR), designed to enhance accuracy and user confidence during aiming tasks by enabling users to sense proximity to and guiding their motion toward a target. To evaluate its effectiveness, we conducted a user study (N = 28) using a custom VR archery simulation, assessing both quantitative performance metrics and qualitative user feedback. With haptic guidance, participants achieved significantly higher targeting accuracy and reported increased spatial awareness, although some also noted a reduced sense of agency. These results suggest that, while haptic assistance offers clear benefits, careful design is essential to avoid overreliance or interference with user autonomy in contexts such as accessibility, gaming, and broader human-computer interaction.
Position-based Dynamics (PBD) algorithms are widely used for simulating deformable models due to their efficiency, robustness and simplicity. However, PBD methods often suffer from the uncontrollable dissipation which leads to highly stiff visual results. Although recent exponential-based frameworks offer practical solutions to this shortcoming, those methods do not provide specific material properties, such as Young's modulus and Poisson's ratio, and implicit volume conservation. To overcome these limitations, we propose to reformulate the potential energy density functions of constitutive materials with an exponential strain tensor for position-based simulation of deformable objects. Our proposed method redefines the well-known potential energy density functions of the linear, corotational, StVK elastic materials and continuous cloth with the exponential strain tensor by conserving their theoretical fundamentals. Therefore, the simulated models provide more vivid visual results against the unexpected numerical dissipation. We compare our method with many well-known methods and demonstrate the advantages. Our method is easy to implement and fits to any existing PBD framework effortlessly.
Games, 3D simulations, and cinematic pipelines depend on realistic 3D terrain for immersion. However, creating detailed 3D terrain is labour-intensive: artists sculpt elevation, iterate on mountains, rivers, lakes, and must often repeat the entire workflow when the design changes. Recent generative approaches are attempting to address this challenge, but they primarily focus on a single landform (typically mountains) and overlook structural features, such as river networks, roads, or lakes. We propose a sketch-conditioned diffusion framework that generates depth maps representing complete landscapes, including mountains, river networks, and lakes. Our method extends Stable Diffusion with a ControlNet branch that takes multiple channel inputs: Canny edges for overall structure, red for mountains, green for lakes, and blue as a carving tool for painting roads and rivers onto the heightmap. This approach addresses the technical challenges while prioritizing the artist's creative control. Our interactive system, Earthbender, gives the artist fine-grained control over every detail in the heightmap, demonstrating a collaborative model where the generative AI acts as a powerful assistant to achieve an artistic vision, rather than replacing the artist's creativity. Our experiments show that our ControlNet-based approach significantly outperforms traditional GANs in both data efficiency and output quality. Furthermore, we present an analysis demonstrating that the choice of loss function acts as a powerful artistic control, allowing the user to select between a sharp, detailed style and a softer, more organic output better suited for downstream game engine workflows.
Creating high-quality character animation remains an intricate and cumbersome process that requires skill, training, and craftsmanship to master. Recently, diffusion models have unlocked the ability to generate diverse movements from high-level condition signals such as text. For artist-friendly control, motion diffusion leveraging Bezier curves have been shown to allow precise joint-level conditioning. Yet, these works have been limited to joints at a fixed temporal stride, while animators require more temporal flexibility when keyframing or manipulating tangents to achieve animation principles such as easing in & out. In this work, we introduce a new Implicit Bezier Motion Model (IBMM), which during training is exposed to all possible configurations of control points, enabling control at arbitrary timings. This allows both precise and sparse joint-level control, anywhere in time and for any joint. In addition, we introduce a new quantitative measure of ease-in and -out, which leads to a novel condition over the motion generation process to reflect this artistic principle.
We present a system for generating expressive, full-body drumming performances from MIDI input, combining rhythmic precision with lifelike motion. Unlike prior work that focuses on limited gestures or audio-driven models, our approach produces coordinated animations of the entire performer, including hands, torso, legs, and facial expressions, driven solely by symbolic MIDI. Our system integrates a Bi-directional LSTM to predict fine-grained 3D hand trajectories, using sticks parented to the hands and synchronized with MIDI events. It also includes a retrieval-based module that generates expressive upper-body and facial motion conditioned on musical phrasing, and a pedal enforcement component that procedurally animates the feet. Our method addresses the unique challenges of drumming, where rhythm is both heard and seen in dynamic, physically grounded motion. To the best of our knowledge, this is the first system to generate full-body drum performances from raw MIDI. Our approach enables new applications in virtual concerts, immersive training, game animation, and digital avatar performance.
This paper presents a time- and space-efficient adaptation of the Space Foundation System, a novel game engine system that abstracts space into semantic locations to bridge the gap between the implementation domain and the problem domain of spatial features in digital games. The presented algorithm discretizes the game space by employing a frontier-based flood fill on a voxel grid to partition the space into locations defined by anchors bounded by delimiters. Integrated into a game engine via extended Transform components, the system enables location queries and supports advanced spatial game features in real time. An evaluation across diverse game environments demonstrates that the algorithm meets practical development requirements regarding computation time and memory usage. The results suggest that the system offers a viable, design-oriented extension to traditional scene graphs.
We introduce a crowd simulation method that uses a block coordinate descent solver reminiscent of projective dynamics approaches. Existing projective dynamics work focuses on simulating deformable materials - which makes us the first to incorporate crowds into such framework. We simulate agents as nodes in a mass-spring system, where springs are created dynamically to simulate different crowd phenomena. We propose novel projective solutions for such springs to emulate time-to-collision avoidance behaviors. Our method encapsulates the collective response of the crowd using implicit Euler integration, while a conjugate gradient solver efficiently resolves the resulting mass-spring linear system to enable the simulation of large-scale, emergent crowd behaviors.
Embodied Conversational Agents (ECAs) are becoming more widely used for various applications, notably in health. Some studies have demonstrated the effectiveness of delivering Motivational Interviews (MIs), a patient-centred behaviour change method, with ECAs. Despite showing promise, the effect of agent communication style on MI effectiveness remains underexplored. Directness has been shown to impact satisfaction and preference of dialogue systems, and effectiveness in some cases. However, outside of verbal style, how to control directness in ECAs to achieve these goals is not well understood. We designed an ECA to convey different levels of directness through language and Non-Verbal behaviours (NVBs), then evaluated the impact of language and NVB on directness with a perception study. The results showed that language influenced perceived directness, and that NVB contributed when aligned with indirect language. These findings suggest ways to shape conversational agents' communication style to enhance their effectiveness.
Artificial intelligence NPCs are known to make games more immersive. However, users' sense of Social Presence with modern, realistic AI NPCs is underexplored. In this paper, we address the research question of how Social Presence is perceived when interacting with AI NPC tutors compared to human tutors in virtual reality assembly tasks. Therefore, we conducted a mixed methods study (N=18) comparing human tutors to AI NPC tutors on a PC assembly task in multiple VR settings with Head-Mounted Displays and a CAVE system. The results indicate that the AI NPC elicits moderate to high levels of Co-Presence and moderate levels of Social Presence, and most participants reported that the AI NPC is beneficial in completing the PC assembly task. Nevertheless, the human tutor was rated significantly higher in terms of Social Presence. We infer suggestions for increasing the Social Presence with AI NPCs, such as increasing the quality of animations and vocal features, important for game research as well as Intelligent Tutoring Systems.
We propose MIRRORED-Anims, a novel retargeting procedure for transferring motion between skinned humanoid characters of different morphologies. It is designed so as to mimic the strengths of the closed-source Mixamo's retargeting method, currently used as a standard to create motion databases and train all state-of-the-art learning-based retargeting methods, despite severe shortcomings (namely, a lack of character diversity and notable penetration artifacts). Taking inspiration from the toolsets of 3D animators, our retargeting algorithm relies on the control rigs used to manipulate skinned characters, by identifying and transferring controller values on predefined bone mechanisms. While producing motions which are closer to Mixamo's ground truth than any state-of-the-art learning-based technique, MIRRORED-Anims creates fewer penetration artifacts than observed in the Mixamo dataset, improving the perceived quality of the output. Moreover, motion can be retargeted in real-time to and from the SMPL body model, making it possible to leverage the large motion databases available in SMPL format for the retargeting task. Because it relies solely on transparent, explainable rig operations, MIRRORED-Anims can be used to generate ground-truth motions for any humanoid character, providing a reliable baseline for the future training of learning-based methods. Project page: https://mirrored-anims.github.io/MIRRORED-Anims
Concerts and performances in virtual reality are becoming increasingly popular. Accurately visualizing the detailed hand motions of musicians playing instruments is challenging. In this work, we present a real-time motion synthesis method that generates the detailed fretting hand motion for playing guitar. Our approach first involves capturing and post-processing hand motion data from guitar performances to create a data set for training. The post-processing ensures that the fingertip positions are placed as accurately as possible regarding distance from the fretboard and location between the frets. We then train a neural network that learns to predict hand and finger poses based on guitar tabs and previous poses. We found that our method produces reasonably stable motion and evaluate our results using accuracy measures and visual evaluation.
Understanding player behavior, movement, and dynamics is crucial for uncovering how players interact with game environments, optimize strategies, and engage in competitive scenarios, offering insights that inform game design, AI development, and human interaction. We present a spatial-temporal analysis of player movement, path trajectories, and behavioral strategies on the Call of Duty: Warzone Caldera map using an open-source telemetry dataset. We characterize player motion through kinematic metrics such as distance, entropy, acceleration, and jerk to capture fine-grained movement dynamics under changing game conditions. Our analysis highlights the interplay and behavioral patterns across skill levels and player roles by comparing the highest and lowest-ranked players, revealing strategies linked to path optimization and survival.
Expressive facial animation depends on models that can convey subtle, context-dependent emotions. Procedural methods often rely on manually-tuned heuristics, while data-driven techniques are constrained by the diversity of their training data. This paper explores the zero-shot potential of large language models (LLMs) to generate facial animations. Using the cognitively-grounded Ortony-Clore-Collins (OCC) model as a framework, we designed 110 text-based scenarios and evaluated the ability of different LLMs - Gemini-2.5 Pro, GPT-4o, and Llama 3.1-8b - to generate corresponding facial animations using the Facial Action Coding System (FACS). A perceptual user study confirmed that animations from Gemini-2.5 Pro were highly recognizable, with participants successfully matching facial expressions to the correct context at rates significantly above chance. A quantitative analysis of the generated Action Units (AUs) indicated both consistency within each OCC emotion category and diversity across different scenarios. Further analysis revealed that the emotional expressions group into six clusters: happiness, sadness, anger/disgust, fear/surprise, shame, and neutral. This work demonstrates a viable, lightweight pipeline connecting textual narrative directly to motion generation without requiring custom training or large-scale motion capture datasets.
We present a fully text-driven framework for 3D facial animation that eliminates the need for audio input or explicit prosodic cues. Our architecture extracts rich phoneme embeddings from text using a pre-trained TTS encoder, aligns them with quantized motion embeddings via a transformer decoder, and decodes the result into mesh deformations through a pre-trained transformer decoder. We explore two scenarios of our pipeline: (1) In the single-subject setting, we find that phoneme embeddings alone can yield accurate lip motion. (2) In a multi-subject setting, where speaker articulation varies widely, we introduce stochastic latent modulation to model residual variability conditioned on both phoneme context and speaker identity. We evaluate our approach quantitatively and qualitatively: We demonstrate accurate lip sync in the single-subject case, and compare against audio-driven baselines on a large multi-subject dataset. Our results show that PhonemeNet not only achieves competitive lip sync and motion quality, but also offers flexibility, modularity, and scalability as an alternative to audio-driven facial animation.
Spatiotemporal sensitivity to high frequency information declines with increased peripheral eccentricity. Foveated rendering exploits this by decreasing the spatial resolution of rendered images in peripheral vision, reducing the rendering cost by omitting high frequency details. As foveation levels increase, the rendering quality is reduced, and traditional foveated rendering systems tend not to preserve samples that were previously rendered at high spatial resolution in previous frames. Additionally, prior research has shown that saccade landing positions are distributed around a target location rather than landing at a single point, and that even during fixations, eyes perform small microsaccades around a fixation point. This creates an opportunity for sampling from temporally neighbouring frames with differing foveal locations to reduce the required rendered size of the foveal region while achieving a higher perceived image quality. We further observe that the temporal presentation of pixels frame-to-frame can be viewed as a data stream, presenting a random sampling problem. Following this intuition, we propose a Weighted Reservoir Sampling technique to efficiently maintain a reservoir of the perceptually relevant high quality pixel samples from previous frames and incorporate them into the computation of the current frame. This allows the renderer to render a smaller region of foveal pixels per frame by temporally reusing pixel samples that are still relevant to reconstruct a higher perceived image quality, while allowing for higher levels of foveation. Our method operates on the output of foveated rendering, and runs in under 1 ms at 4K resolution, making it highly efficient and integrable with real-time VR and AR foveated rendering systems.
Creating human digital doubles is becoming easier and much more accessible to everyone using consumer grade devices. In this work, we investigate how avatar style (realistic vs cartoon) and avatar familiarity (self, acquaintance, unknown person) affect self/other-identification, perceived realism, affinity and social presence with a controlled offline experiment. We created two styles of avatars (realistic-looking MetaHumans and cartoon-looking ReadyPlayerMe avatars) and facial animations stimuli for them using performance capture. Questionnaire responses demonstrate that higher appearance realism leads to a higher level of identification, perceived realism and social presence. However, avatars with familiar faces, especially those with high appearance realism, lead to a lower levels of identification, perceived realism, and affinity. Although participants identified their digital doubles as their own, they consistently did not like their avatars, especially of realistic appearance. But they were less critical and more forgiving about their acquaintance's or an unknown person's digital double.
We present an approach for training "agile" character control policies, able to produce a wide variety of motor skills from a single reference motion cycle. Our technique builds off of generative adversarial imitation learning (GAIL), with a key novelty of our approach being to provide modification to the observation map in order to improve agility and robustness. Namely, to support more agile behavior, we adjust the value measurements of the training discriminator through relative features - hence the name ReGAIL. Our state observations include both task relevant relative velocities and poses, as well as relative goal deviation information. In addition, to increase robustness of the resulting gaits, servo gains and damping values are included as part of the policy action to let the controller learn how to best combine tension and relaxation during motion. From a policy informed by a single reference motion, our resulting agent is able to maneuver as needed, at runtime, from walking forward to walking backward or sideways, turning and stepping nimbly. We demonstrate our approach for a humanoid and a quadruped, on both flat and sloped terrains, as well as provide ablation studies to validate the design choices of our framework.
