Classical neuronal cable theory relies on quasi-static electric field approximations and neglects magnetic induction, Lorentz force coupling, and transient electromagnetic currents, limiting its ability to fully characterize action potential propagation within geometrically branched axons and dendrites. This work develops a coupled Maxwell-electromagnetic cable framework by integrating finite-difference time-domain (FDTD) solutions of Maxwell's equations with extended Hodgkin-Huxley and Fitzhugh-Nagumo membrane dynamics, incorporating magnetic gating perturbations, electromagnetic trans-membrane currents $I_{\text{EM}}$, and nanoscale quantum corrections for thin neural segments. Controlled propagation experiments are designed to quantify deviations from standard cable predictions across asymmetric and symmetric axonal bifurcation geometries. Numerical results demonstrate that inductive magnetic effects lower the critical branch radius for junction conduction failure and break symmetric action potential invasion in geometrically identical child branches under external transverse magnetic fields. An electromagnetic corrected geometric ratio $GR_{\text{EM}}$ is proposed to revise impedance-matching conditions at branch points, accounting for size-dependent axial current imbalance induced by magnetic and displacement currents. Parent axon conduction velocity deviates substantially from the canonical $\sqrt{d}$ scaling law when electromagnetic feedback and quantum charge distributions are included, triggering early signal blockage at large cable diameters. Collectively, this study establishes that quasi-static cable models underestimate electromagnetic corrections to propagation speed, waveform shape, and bifurcation transmission fidelity; the coupled Maxwell-cable framework provides a comprehensive multi-physics tool for modeling electrodynamic signal behavior in complex neuronal architectures.

While autonomous multi-robots can achieve safe and coordinated navigation, they often struggle to adapt to unforeseen conditions and to capture operator-driven objectives in unstructured environments. We present a Virtual Reality (VR)-based shared control framework for teams of drones operating in constrained and unknown environments, enabling real-time, user-guided exploration. Our approach integrates a novel user-guided motion-primitive-based planner with an admittance controller, generating dynamically feasible, collision-free trajectories while allowing the operator to flexibly influence team behavior. By leveraging user input, the framework enables the robot team to explore regions of interest that autonomous planners may overlook. The system supports mixed-reality operations with both physical and simulated drones, and implements a bilateral VR-based interface, allowing the operator to guide the robot team via migration points while receiving immediate visual feedback of the team state. Experimental results show that shared control improves obstacle avoidance, maintains inter-agent spacing, and reduces operator effort, demonstrating the feasibility and advantages of immersive, human-in-the-loop swarm navigation

Shape-changing User Interfaces (SCUIs) can dynamically adjust their shape or layout in response to user interactions or environments. It is challenging to design expandable, affordable, and effective SCUIs with optimal space utilization for novel interactions. To tackle this challenge we introduce Robotecture, a cost-efficient and expandable shape-changing system which utilizes a self-lifting structure composed of modular robotics that actuate support beams. Robotecture generates dynamic surface displays and enclosures by modulating a grid of robotic units with linear movements, each with two actuators and four beams connecting to adjacent units. The modular design allows the structure to scale to different grid sizes and to be arranged in flexible layouts. The self-lifting nature of Robotecture makes it possible to utilize the space on both sides of the surface. The design of a sparse grid structure makes the system more efficient in simulating large-scale structures such as smart architecture, and the spaces between the beams enable objects to pass through the actuated surface for novel interactions. In this paper, we demonstrate a few prototypes with different layouts and validate the proof of concept. Additionally, we showcase various scenarios where Robotecture can enhance tangible interactions and interactive experiences for versatile and eco-friendly applications across different scales, such as tabletop tangible displays, room-scale furniture simulation, smart architecture, etc.

Manipulating large virtual objects in extended reality (XR) environments poses challenges due to scale, occlusion, and interaction precision. This work presents a system design and a comparative study of two interfaces for large-object manipulation: a tangible user interface (TUI) using 3D-printed Opti-Tracked miniatures, and a 2D graphical user interface (GUI) minimap. We deployed the system at SIGGRAPH ASIA 2024 with a floor-planning scenario and collected feedback from over 100 participants. Results show that TUI enhance spatial awareness and collaboration, while GUIs support more focused layout planning with greater precision. Our findings highlight trade-offs between tangible and graphical interfaces, suggesting hybrid approaches for improved usability in large-scale XR environment arrangement tasks.

The 1st International Workshop on Spatial Memory in eXtended Reality (XRMemory) aims to explore and advance technologies that enable immersive, memory-rich experiences, allowing users to revisit moments as if they were truly back in those spaces. While photos and videos have traditionally served this purpose, advancements in Augmented Reality (AR), Virtual Reality (VR), and spatial computing now enable far more realistic and immersive memory recall. This workshop investigates how AI and XR technologies can be harnessed to record, generate, and recall user experiences in transformative ways. By enabling individuals to relive their own memories or immerse themselves in others’ experiences as if physically present, it seeks to open new avenues for shared connections that transcend the limitations of time and space.

Traditional design processes for physical prototypes can be time-consuming and costly due to iterations of prototyping, testing, and refinement. Extended Reality (XR) technology with video passthrough offers unique benefits for alleviating these issues by providing instant visual feedback. We have developed an XR system with multimodal input capability that provides annotations and enables interactive visual modifications by superimposing and aligning visual counterparts to physical objects. This system can help designers to quickly experiment with and visualize a wide range of design options, keep track of design iterations, and explore innovative solutions without the constraints of physical prototyping. As a result, it can significantly speed up the iterative design process, while requiring fewer physical modifications in each iteration.

We introduce DrawTalking, a prototype system enabling an approach that empowers users to build interactive worlds by sketching and speaking. The approach emphasizes user control and flexibility, and gives programming-like capability without requiring code. An early open-ended study shows the mechanics resonate and are applicable to many creative-exploratory use cases, with the potential to inspire and inform research in future natural interfaces for creative exploration and authoring.

Terrain generation and authoring in Virtual Reality (VR) offers unique benefits for terrain authoring including stereo display, immersive and intuitive design experience, and natural input modalities. We present this VR-based terrain fast prototyping system to integrate natural input modalities, preserve artistic controls and lower the effort of landscape prototyping. The system utilizes freehand interfaces and a generative model to help users quickly prototype different types of natural landscapes, such as mountains, mesas, canyons, and volcanoes. With the freehand interfaces, users can use their hands to draw mid-air strokes as the 3D contours of the desired landscapes. Then, a Conditional Generative Adversarial Network (CGAN) generates realistic landscapes based on the 3D contour. The freehand interface detects users’ gestures to control landscape editing. By incorporating CGAN as the terrain synthesizer, our system allows users to immersively, rapidly, and easily prototype terrains using intuitive interactive hand controls.

Terrain generation and authoring in Virtual Reality (VR) offers unique benefits, including 360-degree views, improved spatial perception, immersive and intuitive design experience and natural input modalities. Yet even in VR it can be challenging to integrate natural input modalities, preserve artistic controls and lower the effort of landscape prototyping. To tackle these challenges, we present our VR-based terrain generation and authoring system, which utilizes hand tracking and a generative model to allow users to quickly prototype natural landscapes, such as mountains, mesas, canyons and volcanoes. Via positional hand tracking and hand gesture detection, users can use their hands to draw mid-air strokes to indicate desired shapes for the landscapes. A Conditional Generative Adversarial Network trained by using real-world terrains and their height maps then helps to generate a realistic landscape which combines features of training data and the mid-air strokes. In addition, users can use their hands to further manipulate their mid-air strokes to edit the landscapes. In this paper, we explore this design space and present various scenarios of terrain generation. Additionally, we evaluate our system across a diverse user base that varies in VR experience and professional background. The study results indicate that our system is feasible, user-friendly and capable of fast prototyping.

We present a hydrodynamic simulation system using the GPU compute shaders of DirectX for simulating virtual agent behaviors and navigation inside a smoothed particle hydrodynamical (SPH) fluid environment with real-time water mesh surface reconstruction. The current SPH literature includes interactions between SPH and heterogeneous meshes but seldom involves interactions between SPH and virtual boid agents. The contribution of the system lies in the combination of the parallel smoothed particle hydrodynamics model with the distributed boid model of virtual agents to enable agents to interact with fluids. The agents based on the boid algorithm influence the motion of SPH fluid particles, and the forces from the SPH algorithm affect the movement of the boids. To enable realistic fluid rendering and simulation in a particle-based system, it is essential to construct a mesh from the particle attributes. Our system also contributes to the surface reconstruction aspect of the pipeline, in which we performed a set of experiments with the parallel marching cubes algorithm per frame for constructing the mesh from the fluid particles in a real-time compute and memory-intensive application, producing a wide range of triangle configurations. We also demonstrate that our system is versatile enough for reinforced robotic agents instead of boid agents to interact with the fluid environment for underwater navigation and remote control engineering purposes.

We present "Push-That-There", an interaction method and system enabling multimodel object-level user interaction with multi-robot system to autonomously and collectively manipulate objects on tabletop surfaces, inspired by "Put-That-There". Rather than requiring users to instruct individual robots, users directly specify how they want the objects to be moved, and the system responds by autonomously moving objects via our generalizable multi-robot control algorithm. The system is combined with various user instruction modalities, including gestures, GUI, tangible manipulation, and speech, allowing users to intuitively create object-level instruction. We outline a design space, highlight interaction design opportunities facilitated by "Push-That-There", and provide an evaluation to assess our system's technical capabilities. While other recent HCI research has studied interaction using multi-robot system (e.g. Swarm UIs), our contribution is in the design and technical implementation of intuitive object-level interaction for multi-robot system that allows users to work at a high level, rather than needing to focus on the movements of individual robots.

We introduce DrawTalking, an approach to building and controlling interactive worlds by sketching and speaking while telling stories. It emphasizes user control and flexibility, and gives programming-like capability without requiring code. An early open-ended study with our prototype shows that the mechanics resonate and are applicable to many creative-exploratory use cases, with the potential to inspire and inform research in future natural interfaces for creative exploration and authoring.

We created a clinical virtual reality application for vestibular rehabilitation. Our app targets contextual sensory integration (C.S.I.) where patients are immersed in safe, increasingly challenging environments while practicing various tasks (e.g., turning, walking). The purpose of this pilot study was to establish the feasibility of a randomized controlled trial comparing C.S.I. training to traditional vestibular rehabilitation. Thirty patients with vestibular dysfunction completed the Dizziness Handicap Inventory (DHI), Activities-Specific Balance Confidence Scale (ABC), Visual Vertigo Analog Scale (VVAS), Functional Gait Assessment (FGA), Timed-Up-and-Go (TUG), and Four-Square Step Test (FSST). Following initial assessment, the patients were randomized into 8 weeks (once per week in clinic + home exercise program) of traditional vestibular rehabilitation or C.S.I. training. Six patients had to stop participation due to the covid-19 pandemic, 6 dropped out for other reasons (3 from each group). Ten patients in the traditional group and 8 in the C.S.I group completed the study. We applied an intention to treat analysis. Following intervention, we observed a significant main effect of time with no main effect of group or group by time interaction for the DHI (mean difference − 18.703, 95 https://clinicaltrials.gov/ct2/show/NCT04268745 .

In collocated VR classes, instructors need to guide their students, while also remaining aware of the physical environment in order to ensure students’ safety. It is hard to do both simultaneously. We present a system that utilizes hand-held devices for non-VR instructors, enabling them to explore VR content and interact with students who are fully immersed in VR. The instructor can observe the VR environment or switch between different students’ first-person views by using commonly available hand-held devices, such as smartphones and tablets. The instructor can also use hand-held devices to interact with the VR world itself. The students can see the real-time video stream of the physical environment as well as a video stream of the instructor. The system enables seamless communication and collaboration, thereby helping to create a better and richer educational experience for VR classes.

OBJECTIVE: The purpose of this study was to determine the extent to which sensory integration strategies via head sway, derived from a Head-Mounted Display (HMD), change in people with vestibular disorders following vestibular rehabilitation. DESIGN: Randomized Controlled Trial SETTING: Vestibular Rehabilitation Clinic PARTICIPANTS: Thirty participants with vestibular dysfunction and 21 age-matched controls. MAIN OUTCOME MEASURES: Participants experienced two levels of visual surround (static or moving 'stars', front to back at 0.2 Hz, 32 mm) and white noise (none or rhythmic) while their head sway was recorded via the HTC Vive. We quantified head sway via Directional Path (DP) and Root Mean Square Velocity (RMSV) in 5 directions: anterior-posterior, medio-lateral, pitch, yaw, and roll and Power Spectral Density in low (PSD 1), medium (PSD 2) and high (PSD 3) frequencies in the anterior-posterior direction. INTERVENTIONS: Participants performed the assessment prior to being randomized into 8-weeks of contextual sensory integration training in virtual reality or traditional vestibular rehabilitation and once again following completion of the intervention. Controls performed the assessment once. Twelve participants dropped out, half due to covid lock-down. We applied an intention to treat analysis. RESULTS: We observed significant increases in AP DP, RMSV and all PSDs with change in visual level. Both intervention groups significantly decreased medio-lateral, pitch and roll DP and RMSV and anterior-posterior PSD 2 with no group differences. Vestibular participants were significantly higher than controls on all outcomes pre rehabilitation. Post rehabilitation they were only significantly higher on PSD 2. Sound was not a significant predictor of head sway in this protocol. CONCLUSIONS: Head sway decreased following vestibular rehabilitation regardless of visual load or type of intervention applied. This change was measured via head kinematics derived from a portable HMD which can serve as a sensitive in-clinic assessment for tracking improvement over time

Our project combines immersive VR, multitouch AR, real-time volumetric capture, motion capture, robotically-actuated tangible interfaces at multiple scales, and live coding, in service of a human-centric way of collaborating. Participants bring their unique talents and preferences to collaboratively tackle complex problems in a shared mixed reality world.

How do we design effective immersive VR experiences? Looking Inside: Cells is a set of collaborative immersive virtual reality science learning simulations that were designed by applying best practices for learning experience design, taking advantage of the unique affordances of VR for learning. Simulations were designed with input from teachers and students and cover NGSS-aligned middle school science topics in cell biology, including plant cell, animal cell, and prokaryotic cells and their respective organelles, cell specialization, mitosis, and viral mutation. The interaction design of the simulation implements engagements with the learning materials that support deep learning. The collaboration design allows small groups to work on the simulation together. Emotional design was used to induce emotions that are conducive for learning. Classroom integration features are designed to support teachers’ use of the simulations through lesson plans, teacher professional development, a teacher dashboard, photo taking, and a spectator mode. Ongoing user research provides feedback from teachers and students that is used to refine the design with the goal to enhance learning outcomes.

This pilot study aimed to identify postural strategies in response to sensory perturbations (visual, auditory, somatosensory) in adults with and without sensory loss. We tested people with unilateral peripheral vestibular hypofunction (N = 12, mean age 62 range 23-78), or with Unilateral Sensorineural Hearing Loss (USNHL, N = 9, 48, 22-82), or healthy controls (N = 21, 52, 28-80). Postural sway and head kinematics parameters (Directional Path in the anterior-posterior and medio-lateral directions (sway & head); pitch, yaw and roll (head) were analyzed in response to 2 levels of auditory (none, rhythmic sounds via headphones), visual (static, dynamic) and somatosensory cues (floor, foam) within a simulated, virtual 3-wall display of stars. We found no differences with the rhythmic auditory cues. The effect of foam was magnified in the vestibular group compared with controls for anterior-posterior and medio-lateral postural sway, and all head direction except for medio-lateral. The vestibular group had significantly larger anterior-posterior and medio-lateral postural sway and head movement on the static scene compared with controls. Differences in pitch, yaw and roll emerged between vestibular and controls only with sensory perturbations. The USNHL group did not increase their postural sway and head movement with the increased visual load as much as controls did, particularly when standing on the foam. They did not increase their medio-lateral sway with the foam as much as controls did. These findings suggest that individuals with USNHL employ a compensatory strategy of conscious control of balance, the functional implications of which need to be tested in future research.

In another decade smart glasses, and the networked infrastructure that will make them possible, will fundamentally alter all that we know. It is hard to fully anticipate the impact of such a profound change, but we can make a few predictions.