Visuotactile pretraining with human data leads to robust manipulation policies trained in simulation.

To achieve human-level dexterity, robots must infer spatial awareness from multimodal sensing to reason over contact interactions. During in-hand manipulation of novel objects, such spatial awareness involves estimating the object's pose and shape. The status quo for in-hand perception primarily uses vision and is restricted to tracking a priori known objects. Moreover, visual occlusion of objects in hand is imminent during manipulation, preventing current systems from pushing beyond tasks without occlusion. We combined vision and touch sensing on a multifingered hand to estimate an object's pose and shape during in-hand manipulation. Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem. We studied multimodal in-hand perception in simulation and the real world, interacting with different objects via a proprioception-driven policy. Our experiments showed final reconstruction F scores of 81% and average pose drifts of 4.7 millimeters, which was further reduced to 2.3 millimeters with known object models. In addition, we observed that, under heavy visual occlusion, we could achieve improvements in tracking up to 94% compared with vision-only methods. Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation. We release our evaluation dataset of 70 experiments, FeelSight, as a step toward benchmarking in this domain. Our neural representation driven by multimodal sensing can serve as a perception backbone toward advancing robot dexterity.

We introduce RotateIt, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs. Our system is trained in simulation, where it has access to ground-truth object shapes and physical properties. Then we distill it to operate on realistic yet noisy simulated visuotactile and proprioceptive sensory inputs. These multimodal inputs are fused via a visuotactile transformer, enabling online inference of object shapes and physical properties during deployment. We show significant performance improvements over prior methods and the importance of visual and tactile sensing.

To achieve human-level dexterity, robots must infer spatial awareness from multimodal sensing to reason over contact interactions. During in-hand manipulation of novel objects, such spatial awareness involves estimating the object's pose and shape. The status quo for in-hand perception primarily employs vision, and restricts to tracking a priori known objects. Moreover, visual occlusion of objects in-hand is imminent during manipulation, preventing current systems to push beyond tasks without occlusion. We combine vision and touch sensing on a multi-fingered hand to estimate an object's pose and shape during in-hand manipulation. Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem. We study multimodal in-hand perception in simulation and the real-world, interacting with different objects via a proprioception-driven policy. Our experiments show final reconstruction F-scores of 81 pose drifts of 4.7 mm, further reduced to 2.3 mm with known CAD models. Additionally, we observe that under heavy visual occlusion we can achieve up to 94 Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation. We release our evaluation dataset of 70 experiments, FeelSight, as a step towards benchmarking in this domain. Our neural representation driven by multimodal sensing can serve as a perception backbone towards advancing robot dexterity. Videos can be found on our project website https://suddhu.github.io/neural-feels/

We present MidasTouch, a tactile perception system for online global localization of a vision-based touch sensor sliding on an object surface. This framework takes in posed tactile images over time, and outputs an evolving distribution of sensor pose on the object's surface, without the need for visual priors. Our key insight is to estimate local surface geometry with tactile sensing, learn a compact representation for it, and disambiguate these signals over a long time horizon. The backbone of MidasTouch is a Monte-Carlo particle filter, with a measurement model based on a tactile code network learned from tactile simulation. This network, inspired by LIDAR place recognition, compactly summarizes local surface geometries. These generated codes are efficiently compared against a precomputed tactile codebook per-object, to update the pose distribution. We further release the YCB-Slide dataset of real-world and simulated forceful sliding interactions between a vision-based tactile sensor and standard YCB objects. While single-touch localization can be inherently ambiguous, we can quickly localize our sensor by traversing salient surface geometries. Project page: https://suddhu.github.io/midastouch-tactile/

Knowledge of 3-D object shape is of great importance to robot manipulation tasks, but may not be readily available in unstructured environments. While vision is often occluded during robot-object interaction, high-resolution tactile sensors can give a dense local perspective of the object. However, tactile sensors have limited sensing area and the shape representation must faithfully approximate non-contact areas. In addition, a key challenge is efficiently incorporating these dense tactile measurements into a 3-D mapping framework. In this work, we propose an incremental shape mapping method using a GelSight tactile sensor and a depth camera. Local shape is recovered from tactile images via a learned model trained in simulation. Through efficient inference on a spatial factor graph informed by a Gaussian process, we build an implicit surface representation of the object. We demonstrate visuo-tactile mapping in both simulated and real-world experiments, to incrementally build 3-D reconstructions of household objects.

Tactile perception is central to robot manipulation in unstructured environments. However, it requires contact, and a mature implementation must infer object models while also accounting for the motion induced by the interaction. In this work, we present a method to estimate both object shape and pose in real-time from a stream of tactile measurements. This is applied towards tactile exploration of an unknown object by planar pushing. We consider this as an online SLAM problem with a nonparametric shape representation. Our formulation of tactile inference alternates between Gaussian process implicit surface regression and pose estimation on a factor graph. Through a combination of local Gaussian processes and fixed-lag smoothing, we infer object shape and pose in real-time. We evaluate our system across different objects in both simulated and real-world planar pushing tasks.

In this paper, we introduce an ambiguity-aware robust active SLAM (ARAS) framework that makes use of multi-hypothesis state and map estimations to achieve better robustness. Ambiguous measurements can result in multiple probable solutions in a multi-hypothesis SLAM (MH-SLAM) system if they are temporarily unsolvable (due to insufficient information), our ARAS aims at taking all these probable estimations into account explicitly for decision making and planning, which, to the best of our knowledge, has not yet been covered by any previous active SLAM approach (which mostly consider a single hypothesis at a time). This novel ARAS framework 1) adopts local contours for efficient multi-hypothesis exploration, 2) incorporates an active loop closing module that revisits mapped areas to acquire information for hypotheses pruning to maintain the overall computational efficiency, and 3) demonstrates how to use the output target pose for path planning under the multi-hypothesis estimations. Through extensive simulations and a real-world experiment, we demonstrate that the proposed ARAS algorithm can actively map general indoor environments more robustly than a similar single-hypothesis approach in the presence of ambiguities.

Robots need accurate, online estimates of shape and pose while manipulating unknown objects. While vision and depth-based tracking has been well studied [3], they are affected by self-occlusion, cluttered workspaces, and poor visibility. Interestingly—even when blindfolded—humans can infer object properties from local tactile information. Online tactile inference is hard due to the intrusive nature of touch sensing, and initially unknown object models. Recently, Yu et al. [1] formulated this as a batch-SLAM problem, relying on frictional pushing mechanics. However, the method is not built for online tracking, and uses a piecewise-linear discrete shape representation. Incremental, graph-based approaches were later considered, but assume known object model and incorporate vision [2, 4]. A Gaussian process implicit surface (GPIS) shape representation can fuse uncertain measurements in a probabilistic fashion, and is non-parametric—unlike [1]. Dragiev et al. [5] use a GPIS for tactile exploration of fixed pose 3-D objects. To our knowledge, no methods use this representation with online pose estimation for manipulation tasks. In this work, we combine a GPIS shape representation with sparse nonlinear incremental optimization to localize and infer the shape of planar objects. We demonstrate results with simulated tactile exploration of different objects.

In this paper, we present an active SLAM framework for volumetric exploration of 3D underwater environments with multibeam sonar. Recent work in integrated SLAM and planning performs localization while maintaining volumetric free-space information. However, an absence of informative loop closures can lead to imperfect maps, and therefore unsafe behavior. To solve this, we propose a navigation policy that reduces vehicle pose uncertainty by balancing between volumetric exploration and revisitation. To identify locations to revisit, we build a 3D visual dictionary from real-world sonar data and compute a metric of submap saliency. Revisit actions are chosen based on propagated pose uncertainty and sensor information gain. Loop closures are integrated as constraints in our pose-graph SLAM formulation and these deform the global occupancy grid map. We evaluate our performance in simulation and real-world experiments, and highlight the advantages over an uncertainty-agnostic framework.

In this letter, we propose a novel method for underwater localization using natural visual landmarks above the water surface. High-accuracy, drift-free pose estimates are necessary for inspection tasks in underwater indoor environments, such as nuclear spent pools. Inaccuracies in robot localization degrade the quality of its obtained map. Our framework uses sparse features obtained via an onboard upward-facing stereo camera to build a global ceiling feature map. However, adopting the pinhole camera model without explicitly modeling light refraction at the water-air interface contributes to a systematic error in observations. Therefore, we use refraction-corrected projection and triangulation functions to obtain true landmark estimates. The SLAM framework jointly optimizes vehicle odometry and point landmarks in a global factor graph using an incremental smoothing and mapping backend. To the best of our knowledge, this is the first method that observes in-air landmarks through water for underwater localization. We evaluate our method via both simulation and real-world experiments in a test-tank environment. The results show accurate localization across various challenging scenarios.

Autonomous underwater vehicles have the potential to inspect and map indoor underwater environments, such as spent nuclear fuel pools and ship ballast tanks. These environments are to be regularly monitored for structural integrity—existing manual methods are expensive, dangerous and slow. Employing an autonomous agent presents distinct challenges in SLAM and exploration. This thesis makes contributions in the domains of visual localization and active SLAM for these environments. First, we propose a novel through-water method for visual localization using landmarks above the water surface. With dead-reckoning, the vehicle pose estimate drifts and the errors propagate to the resultant map. Adopting methods from multimedia photogrammetry in our localization framework, we model refraction at the water-air interface. To the best of our knowledge, this is the first through-water method for underwater localization. We evaluate our method via both simulation and real-world experiments in a test-tank environment. The second work presents an active SLAM framework for sonar mapping of these environments. Accurate mapping requires jointly considering the robot trajectory with the state estimation problem. Building on previous work in mapping and planning, we devise an exploration policy that bounds pose uncertainty through revisit actions. A revisit policy is selected based on submap saliency, propagated pose uncertainty, and path information gain. We demonstrate the system in simulation and highlight the advantages over an uncertainty-agnostic framework.

In this paper we address the task of determining the geographical location of an image, a pertinent problem in learning and computer vision. This research was inspired from playing GeoGuessr, a game that tests a humans' ability to localize themselves using just images of their surroundings. In particular, we wish to investigate how geographical, ecological and man-made features generalize for random location prediction. This is framed as a classification problem: given images sampled from the USA, the most-probable state among 50 is predicted. Previous work uses models extensively trained on large, unfiltered online datasets that are primed towards specific locations. To this end, we create (and open-source) the 50States10K dataset - with 0.5 million Google Street View images of the country. A deep neural network based on the ResNet architecture is trained, and four different strategies of incorporating low-level cardinality information are presented. This model achieves an accuracy 20 times better than chance on a test dataset, which rises to 71.87% when taking the best of top-5 guesses. The network also beats human subjects in 4 out of 5 rounds of GeoGuessr.

Current methods for inspection of spent nuclear fuel storage basins involve lowering a single camera for visual inspection of walls and other structures. We present a localized inspection solution where the images are automatically annotated by localization information and a 3D model of the inspected area is generated. The system consists of an underwater sensor pod containing a stereo pair of cameras, light source, inertial measurement unit, and a pressure sensor. The sensors are time synchronized to provide precise measurements. We describe both the sensor pod and the algorithms that keep the pod localized. Preliminary results from in-air and underwater testing of a prototype are presented.

The study of eye gaze fixations on photographic images is an active research area. In contrast, the image sub-category of freehand sketches has not received as much attention for such studies. In this paper, we analyze the results of a free-viewing gaze fixation study conducted on 3904 freehand sketches distributed across 160 object categories. Our analysis shows that fixation sequences exhibit marked consistency within a sketch, across sketches of a category and even across suitably grouped sets of categories. This multi-level consistency is remarkable given the variability in depiction and extreme image content sparsity that characterizes hand-drawn object sketches. In this paper, we show that the multi-level consistency in the fixation data can be exploited to 1) predict a test sketch’s category given only its fixation sequence and 2) build a computational model which predicts part-labels underlying fixations on objects. We hope that our findings motivate the community to deem sketch-like representations worthy of gaze-based studies vis-a-vis photographic images.

Knowledge of the kinematic state of rovers is critical to navigation, path reconstruction and exploration, especially on rugged terrain like planetary surfaces. Existing methods employ many encoders, potentiometers and hall sensors. These add components and wiring to moving parts. The components are susceptible to mechanical and electronic failures, add mass, and some require thermal regulation. In addition, the sensor wires are susceptible to bending, flexing and wear. Where miniaturization counts, the limitations on mass, size and power encourage elimination of sensors wherever possible. This paper presents a method to estimate the kinematic state of rovers using only a downward-facing fisheye camera. This novel approach implements a vision algorithm to obtain kinematic state information in planetary rovers. The two additional benefits of the technique are (1) redundancy to proprioceptive measurements, (2) means for perceptive visual odometry. The method uses a single camera to estimate 10 degrees-of-freedom (associated with steering, driving and suspension) on the AutoKrawler, a rover test platform for planetary exploration. Motions are estimated by self-perception combining fiducial marker tracking, optical flow techniques and the kinematic constraints of rover mechanics. Experimental results, obtained from the rover operating in an environment analogous to the lunar surface, are presented. The results obtained are compared with ground truth data to validate the approach.