A method to align real and virtual objects for mixed reality investigations of visually guide grasping

Virtual Reality (VR) is a powerful tool for studying perception, allowing researchers to present stimuli whose appearance and physical behaviour deviate from reality. Here, we used VR to investigate visually-guided grasping in mixed reality environments. By aligning real and virtual environments, participants could view virtual versions of objects while touching their real counterparts at the same location. We asked participants to grasp the objects, while we tracked markers attached to their hands and to the objects using 8 Qualisys Miqus M5 motion-tracking cameras. Additionally, we recorded temporally aligned videos using 6 Qualisys Miqus Video cameras. Marker positions were streamed into a virtual environment containing a realistic replica of the real object, which matched the motion of the real object. To verify the alignment between real and virtual scenes, we rendered the virtual environment from the same viewpoints as the video cameras. For visualization purposes, we overlaid the virtual object renderings onto the video recordings, thus generating a new, mixed reality depiction of the scene. To quantify the alignment overlap, we computed the intersection over union of the object segmented out of video and rendered scenes, achieving an average alignment of 84±3 %. We also quantified the alignment accuracy as the distance in image space between real and rendered tracked object markers. We achieved an average accuracy of 6.5±0.5 pixels, which translated to a misalignment of 1.7±0.1 mm and 5.1±0.3 minutes of visual angle for an average viewing distance of 115 cm, validating the effectiveness of our alignment procedure. By rendering the aligned virtual objects with different textures, we can vary the appearance of the objects’ material composition. This allows us to perform novel explorations investigating how the visuomotor system adapts to conflicts between the visual appearance and physical properties of graspable objects.