Tracking passive magnetic markers plays a vital role in advancing healthcare and robotics, offering the potential to significantly improve the precision and efficiency of systems. This technology is key to developing smarter, more responsive tools and devices, such as enhanced surgical instruments, precise diagnostic tools, and robots with improved environmental interaction capabilities. However, traditionally, the tracking of magnetic markers is computationally expensive due to the requirement for iterative optimization procedures. Moreover, these methods depend on the magnetic dipole model for their optimization function, which can yield imprecise outcomes due to the model's significant inaccuracies when dealing with short distances between non-spherical magnet and sensor. Our article introduces a novel approach that leverages neural networks (NNs) to bypass these limitations, directly inferring the marker's position and orientation to accurately determine the magnet's five degrees of freedom (5 DoFs) in a single step without initial estimation. Although our method demands an extensive supervised training phase, we mitigate this by introducing a computationally more efficient method to generate synthetic, yet realistic data using Finite Element Methods simulations. Our novel method uses the rotational symmetry of axis-symmetric magnetic markers to transform the 3-D simulations into 2-D. The benefits of fast and accurate inference significantly outweigh the offline training preparation. In our evaluation, we use different cylindrical magnets, tracked with a square array of 16 sensors. We perform the sensors' reading and position inference on a portable, NN-oriented single-board computer, ensuring a compact setup. We benchmark our prototype against vision-based ground-truth data, achieving a mean positional error of 4 mm and an orientation error of 8 degrees within a $0.2\times 0$ . $2\times 0$ .15 m working volume. These results showcase our prototype's ability to balance accuracy and compactness effectively in tracking 5 DoFs.

The studies of human clothing for digital avatars have predominantly relied on synthetic datasets. While easy to collect, synthetic data often fall short in realism and fail to capture authentic clothing dynamics. Addressing this gap, we introduce 4D-DRESS, the first real-world 4D dataset advancing human clothing research with its high-quality 4D textured scans and garment meshes. 4D-DRESS captures 64 outfits in 520 human motion sequences, amounting to 78k textured scans. Creating a real-world clothing dataset is challenging, particularly in annotating and segmenting the extensive and complex 4D human scans. To address this, we develop a semi-automatic 4D human parsing pipeline. We efficiently combine a human-in-the-loop process with automation to accurately label 4D scans in diverse garments and body movements. Leveraging precise annotations and high-quality garment meshes, we establish several benchmarks for clothing simulation and reconstruction. 4D-DRESS offers realistic and challenging data that complements synthetic sources, paving the way for advancements in research of lifelike human clothing.

We present X-Avatar, a novel avatar model that captures the full expressiveness of digital humans to bring about life-like experiences in telepresence, AR/VR and beyond. Our method models bodies, hands, facial expressions and appearance in a holistic fashion and can be learned from either full 3D scans or RGB-D data. To achieve this, we propose a part-aware learned forward skinning module that can be driven by the parameter space of SMPL-X, allowing for expressive animation of X-Avatars. To efficiently learn the neural shape and deformation fields, we propose novel part-aware sampling and initialization strategies. This leads to higher fidelity results, especially for smaller body parts while maintaining efficient training despite increased number of articulated bones. To capture the appearance of the avatar with high-frequency details, we extend the geometry and deformation fields with a texture network that is conditioned on pose, facial expression, geometry and the normals of the deformed surface. We show experimentally that our method outperforms strong baselines both quantitatively and qualitatively on the animation task. To facilitate future research on expressive avatars we contribute a new dataset, called X-Humans, containing 233 sequences of high-quality textured scans from 20 participants, totalling 35,500 data frames. Project page: https://ait.ethz.ch/X-Avatar.

We present EMDB, the Electromagnetic Database of Global 3D Human Pose and Shape in the Wild. EMDB is a novel dataset that contains high-quality 3D SMPL pose and shape parameters with global body and camera trajectories for in-the-wild videos. We use body-worn, wireless electromagnetic (EM) sensors and a hand-held iPhone to record a total of 58 minutes of motion data, distributed over 81 indoor and outdoor sequences and 10 participants. Together with accurate body poses and shapes, we also provide global camera poses and body root trajectories. To construct EMDB, we propose a multi-stage optimization procedure, which first fits SMPL to the 6-DoF EM measurements and then refines the poses via image observations. To achieve high-quality results, we leverage a neural implicit avatar model to reconstruct detailed human surface geometry and appearance, which allows for improved alignment and smoothness via a dense pixel-level objective. Our evaluations, conducted with a multi-view volumetric capture system, indicate that EMDB has an expected accuracy of 2.3 cm positional and 10.6 degrees angular error, surpassing the accuracy of previous in-the-wild datasets. We evaluate existing state-of-the-art monocular RGB methods for camera-relative and global pose estimation on EMDB. EMDB is publicly available under https://ait.ethz.ch/emdb.

We propose Hi4D, a method and dataset for the automatic analysis of physically close human-human interaction under prolonged contact. Robustly disentangling several in-contact subjects is a challenging task due to occlusions and complex shapes. Hence, existing multi-view systems typically fuse 3D surfaces of close subjects into a single, connected mesh. To address this issue we leverage i) individually fitted neural implicit avatars; ii) an alternating optimization scheme that refines pose and surface through periods of close proximity; and iii) thus segment the fused raw scans into individual instances. From these instances we compile Hi4D dataset of 4D textured scans of 20 subject pairs, 100 sequences, and a total of more than 11 K frames. Hi4D contains rich interaction-centric annotations in 2D and 3D alongside accurately registered parametric body models. We define varied human pose and shape estimation tasks on this dataset and provide results from state-of-the-art methods on these benchmarks. Hi4D dataset can be found at https://ait.ethz.ch/Hi4D.

Tracking passive magnetic markers plays a vital role in advancing healthcare and robotics, offering the potential to significantly improve the precision and efficiency of systems. This technology is key to developing smarter, more responsive tools and devices, such as enhanced surgical instruments, precise diagnostic tools, and robots with improved environmental interaction capabilities. However, traditionally, the tracking of magnetic markers is computationally expensive due to the requirement for iterative optimization procedures. Moreover, these methods depend on the magnetic dipole model for their optimization function, which can yield imprecise outcomes due to the model's significant inaccuracies when dealing with short distances between non-spherical magnet and sensor.Our paper introduces a novel approach that leverages neural networks to bypass these limitations, directly inferring the marker's position and orientation to accurately determine the magnet's 5 DoF in a single step without initial estimation. Although our method demands an extensive supervised training phase, we mitigate this by introducing a computationally more efficient method to generate synthetic, yet realistic data using Finite Element Methods simulations. The benefits of fast and accurate inference significantly outweigh the offline training preparation. In our evaluation, we use different cylindrical magnets, tracked with a square array of 16 sensors. We perform the sensors' reading and position inference on a portable, neural networks-oriented single-board computer, ensuring a compact setup. We benchmark our prototype against vision-based ground truth data, achieving a mean positional error of 4 mm and an orientation error of 8 degrees within a 0.2x0.2x0.15 m working volume. These results showcase our prototype's ability to balance accuracy and compactness effectively in tracking 5 DoF.

To transmit tactile information with a discrete set of symbols, the alphabet of the code has to pack as many stimuli as can be reliably discriminated. Here we introduce a notion of perceptual space to construct such a code, defined in terms of the probabilities that pairs of neighboring stimuli are perceived as equal. We perform discrimination experiments exhibiting the well-known effects described by Weber's law, and identify the physical properties of vibrotactile stimuli that most efficiently modulate perceptual discriminability. We conclude that widely used libraries of vibration profiles contain stimuli that are far from homogeneously discriminable. We present our preliminary results as a starting point for the construction of optimal codes.

We present Hedgehog, a single-actuator spherical pin-array device that produces cutaneous haptic sensations to the user's palms.Hedgehog can enrich digital experiences by providing dynamic haptic patterns over a spherical surface using a simple, hand-held device.The key to our design is that it uses a single central actuator, a spherical omnidirectional electromagnet, to control the extension of all the 86 movable pins.This keeps our design simple to fabricate and scalable.A core challenge with this type of design is that the pins in the array, made out of permanent magnets, need to have a stable position when retracted.We present a method to compute such an arrays' spatial stability, evaluate our hardware implementation in terms of its output force and pin's extension and compare it against our method's predictions.We also report our findings from three user studies investigating the perceived force and speed of traveling patterns.Finally, we present insights on the possible applications of Hedgehog.

We introduce an optimal control method for electromagnetic haptic guidance systems. Our real-time approach assists users in pen-based tasks such as drawing, sketching or designing. The key to our control method is that it guides users, yet does not take away agency. Existing approaches force the stylus to a continuously advancing setpoint on a target trajectory, leading to undesirable behavior such as loss of haptic guidance or unintended snapping. Our control approach, in contrast, gently pulls users towards the target trajectory, allowing them to always easily override the system to adapt their input spontaneously and draw at their own speed. To achieve this flexible guidance, our optimization iteratively predicts the motion of an input device such as a pen, and adjusts the position and strength of an underlying dynamic electromagnetic actuator accordingly. To enable real-time computation, we additionally introduce a novel and fast approximate model of an electromagnet. We demonstrate the applicability of our approach by implementing it on a prototypical hardware platform based on an electromagnet moving on a bi-axial linear stage, as well as a set of applications. Experimental results show that our approach is more accurate and preferred by users compared to open-loop and time-dependent closed-loop approaches.

In this paper we introduce a novel contact-free volumetric haptic feedback device. A symmetric electromagnet is used in combination with a dipole magnet model and a simple control law to deliver dynamically adjustable forces onto a hand-held tool. The tool only requires an embedded permanent magnet and thus can be entirely untethered. The force, however, while contact-free, remains grounded via the spherical electromagnet and relatively large forces (1N at contact) can be felt by the user. The device is capable of rendering both attracting and repulsive forces in a thin shell around the electromagnet. We report findings from a user experiment with 6 participants, characterizing force delivery aspects and perceived precision of our system. We found that users can discern at least 25 locations for repulsive forces.

We present Omni, a self-contained 3D haptic feedback system that is capable of sensing and actuating an untethered, passive tool containing only a small embedded permanent magnet. Omni enriches AR, VR and desktop applications by providing an active haptic experience using a simple apparatus centered around an electromagnetic base. The spatial haptic capabilities of Omni are enabled by a novel gradient-based method to reconstruct the 3D position of the permanent magnet in midair using the measurements from eight off-the-shelf hall sensors that are integrated into the base. Omni's 3 DoF spherical electromagnet simultaneously exerts dynamic and precise radial and tangential forces in a volumetric space around the device. Since our system is fully integrated, contains no moving parts and requires no external tracking, it is easy and affordable to fabricate. We describe Omni's hardware implementation, our 3D reconstruction algorithm, and evaluate the tracking and actuation performance in depth. Finally, we demonstrate its capabilities via a set of interactive usage scenarios.

We demonstrate a system to deliver dynamic guidance in drawing, sketching and handwriting tasks via an electromagnet moving underneath a high refresh rate pressure sensitive tablet presented in \citelangerak2019dynamic. The system allows the user to move the pen at their own pace and style and does not take away control. Using a closed-loop time-free approach allows for error-correcting behavior. The user will experience to be smoothly and natural pulled back to the desired trajectory rather than pushing or pulling the pen to a continuously advancing setpoint. The optimization of the setpoint with regard to the user is unique in our approach.

We introduce TacTiles, light (1.8g), low-power (130 mW), and small form-factor (1 cm 3 ) electromagnetic actuators that can form a flexible haptic array to provide localized tactile feedback. Our novel hardware design uses a custom 8-layer PCB, dampening materials, and asymmetric latching, enabling two distinct modes of actuation: contact and pulse mode. We leverage these modes in Virtual Reality (VR) to render continuous contact with objects and the exploration of object surfaces and volumes with spatial haptic patterns. Results from a series of experiments show that users are able to localize feedback, discriminate between modes with high accuracy, and differentiate objects from haptic surfaces and volumes even without looking at them.

Uncooled infrared detectors have enabled the rapid growth of thermal imaging applications. These detectors are predominantly bolometers, reading out a pixel’s temperature change due to infrared radiation as a resistance change. Another uncooled sensing method is to transduce the infrared radiation into the frequency shift of a mechanical resonator. We present here highly sensitive resonant infrared sensors, based on thermo-responsive shape memory polymers. By exploiting the phase-change polymer as transduction mechanism, our approach provides 2 orders of magnitude improvement of the temperature coefficient of frequency. Noise equivalent temperature difference of 22 mK in vacuum and 112 mK in air are obtained using f/2 optics. The noise equivalent temperature difference is further improved to 6 mK in vacuum by using high-Q silicon nitride membranes as substrates for the shape memory polymers. This high performance in air eliminates the need for vacuum packaging, paving a path towards flexible non-hermetically sealed infrared sensors.

Background The estimation of relative distance is a perceptual task used extensively in everyday life. This important skill suffers from biases that may be more pronounced when estimation is based on haptics. This is especially true for the blind and visually impaired, for which haptic estimation of distances is paramount but not systematically trained. We investigated whether a programmable tactile display, used autonomously, can improve distance discrimination ability in blind and severely visually impaired youngsters between 7 and 22 years-old. Methods Training consisted of four weekly sessions in which participants were asked to haptically find, on the programmable tactile display, the pairs of squares which were separated by the shortest and longest distance in tactile images with multiple squares. A battery of haptic tests with raised-line drawings was administered before and after training, and scores were compared to those of a control group that did only the haptic battery, without doing the distance discrimination training on the tactile display. Results Both blind and severely impaired youngsters became more accurate and faster at the task during training. In haptic battery results, blind and severely impaired youngsters who used the programmable display improved in three and two tests, respectively. In contrast, in the control groups, the blind control group improved in only one test, and the severely visually impaired in no tests. Conclusions Distance discrimination skills can be trained equally well in both blind and severely impaired participants. More importantly, autonomous training with the programmable tactile display had generalized effects beyond the trained task. Participants improved not only in the size discrimination test but also in memory span tests. Our study shows that tactile stimulation training that requires minimal human assistance can effectively improve generic spatial skills.

We present MagTics, a novel flexible and wearable haptic interface based on magnetically actuated bidirectional tactile pixels (taxels). MagTics' thin form factor and flexibility allows for rich haptic feedback in mobile settings. We propose a novel actuation mechanism based on bistable electromagnetic latching that combines high frame rate and holding force with low energy consumption and a soft and flexible form factor. We overcome limitations of traditional soft actuators by placing several hard actuation cells, driven by flexible printed electronics, in a soft 3D printed case. A novel EM-shielding prevents magnet-magnet interactions and allows for high actuator densities. A prototypical implementation comprising of 4 actuated pins on a 1.7 cm pitch, with 2 mm travel, and generating 160 mN to 200 mN of latching force is used to implement a number of compelling application scenarios including adding haptic and tactile display capabilities to wearable devices, to existing input devices and to provide localized haptic feedback in virtual reality. Finally, we report results of a psychophysical study, conducted to inform future developments and to identify possible application domains.

We report the first high-resolution flexible haptic display with 768 (32x24) individually addressable taxels (tactile pixels) designed for wearables and virtual reality (VR) applications. The device integrates a thin Shape Memory Polymer (SMP) membrane with a matrix of compliant carbon-silicone composite heaters, a 4-layer flexible PCB and a flexible fluidic chamber. The actuator yield is 99 %, the taxel pitch is 4 mm and the average displacement is 275 μm with a 225 mN holding force, allowing easy discrimination using the sense of fine touch. One line can be reconfigured and latched in 2.5 s; the entire array can be refreshed in under 1 min 30 s with our current drive circuit. The bistable nature of SMPs enables selective and independent actuator motion by judiciously synchronizing their local Joule heating with a global external pressure supply.

We present MagTics, a novel flexible and wearable haptic interface based on magnetically actuated bidirectional tactile pixels (taxels). MagTics' thin form factor and flexibility allows for rich haptic feedback in mobile settings. We propose a novel actuation mechanism based on bistable electromagnetic latching that combines high frame rate and holding force with low energy consumption and a soft and flexible form factor. We overcome limitations of traditional soft actuators by placing several hard actuation cells, driven by flexible printed electronics, in a soft 3D printed case. A novel EM-shielding prevents magnet-magnet interactions and allows for high actuator densities. A prototypical implementation comprising of 4 actuated pins on a 1.7 cm pitch, with 2 mm travel, and generating 160 mN to 200 mN of latching force is used to implement a number of compelling application scenarios including adding haptic and tactile display capabilities to wearable devices, to existing input devices and to provide localized haptic feedback in virtual reality. Finally, we report results of a psychophysical study, conducted to inform future developments and to identify possible application domains.