Proton: A Visuo-Haptic Data Acquisition System For Robotic Learning Of Surface Properties

Autonomous robots need to efficiently walk over varied surfaces and grasp diverse objects. We hypothesize that the association between how such surfaces look and how they physically react during contact can be learned from a database of matched haptic and visual data recorded from various end-effectors' interactions with thousands of real-world surfaces, such as wood flooring, upholstered fabric, asphalt, grass, and anodized aluminum. As the first step in this effort, we detail the design and construction of the Proton, a multimodal data acquisition system that a human operator can use to gather the envisioned data set. Its sensory modalities include RGBD vision, egomotion, contact force, and contact vibration. Three interchangeable end-effectors (a SynTouch BioTac artificial fingertip, an OptoForce three-axis force sensor, and a steel tooling ball) allow for different material properties at the contact point and provide additional tactile data. This sensor suite emulates the capabilities of the human senses of vision and touch, with the goal of learning surface classification methods that are robust over different sensory modalities. We detail the calibration process for the motion and force sensing systems, as well as a proof-of-concept surface discrimination experiment using the tooling ball end-effector and a Vicon motion tracker. A multi-class SVM trained on the collected force and vibration data achieved 82.5% classification accuracy among five sample surfaces.