Validating the integrity of pipes is an important task for safe natural gas production and many other operations (e.g. refineries, sewers, etc.). Indeed, there is a growing industry of actuated, actively driven mobile robots that are used to inspect pipes. Many rely on a remote operator to inspect data from a fisheye camera to perform manual inspection and provide no localization or mapping capability. In this work, we introduce a visual odometry-based system using calibrated fisheye imagery and sparse structured lighting to produce high-resolution 3D textured surface models of the inner pipe wall. Our work extends state-of-the-art visual odometry and mapping for fisheye systems to incorporate weak geometric constraints based on prior knowledge of the pipe components into a sparse bundle adjustment framework. These constraints prove essential for obtaining high-accuracy solutions given the limited spatial resolution of the fisheye system and challenging raw imagery. We show that sub-millimeter resolution modeling is viable even in pipes which are 400 mm (16”) in diameter, and that sparse range measurements from a structured lighting solution can be used to avoid the inevitable monocular scale drift. Our results show that practical, high-accuracy pipe mapping from a single fisheye camera is within reach.

High-mobility walking robots offer unique capabilities in complex off-road environments where wheeled vehicles are not able to travel. However, these environments can also pose significant autonomous navigation challenges. Key steps in planning a safe path for the robot autonomously include estimating the height of the support ground surface - which is often occluded by vegetation - and classifying the terrain and obstacles above the ground surface. This paper describes the development and experimental evaluation of a terrain classification and ground surface height estimation system to support autonomous navigation for a high-mobility walking robot. We provide experimental evaluation on an extensive, manually-labeled dataset collected from geographically diverse sites over a 28-month period.

A key challenge of developing robots that work closely with people is creating a user interface that allows a user to communicate complex instructions to a robot quickly and easily. We consider a walking logistics support robot, which is designed to carry heavy loads to locations that are too difficult to reach with a wheeled or tracked vehicle. In this application the robot is carrying equipment and supplies for a group of pedestrians, and the primary task for the user interface is to keep the robot traveling with the overall group in the right formation. This paper presents a marker tracking system that uses near infrared cameras, retro-reflective markers, and LIDAR to allow a particular user to designate himself as the robot's leader, and guide the robot along a desired path. We provide an extensive quantitative evaluation to show that the proposed system is able to detect and track a leader through unconstrained and cluttered off-road environments under a wide variety of illumination and motion conditions.

We present a vision-based mapping and localization system for operations in pipes such as those found in Liquified Natural Gas (LNG) production. A forward facing, fisheye camera mounted on a prototype robot collects imagery as it is tele-operated through a pipe network. The images are processed offline to estimate camera pose and sparse scene structure where the results can be used to generate 3D renderings of the pipe surface. The method extends state of the art visual odometry and mapping for fisheye systems to incorporate geometric constraints based on prior knowledge of the pipe components into a Sparse Bundle Adjustment framework. These constraints significantly reduce inaccuracies resulting from the limited spatial resolution of the fisheye imagery, limited image texture, and visual aliasing. Preliminary results are presented for a dataset collected in fiberglass pipe network which demonstrate the validity of the approach.

We present a vision-based mapping and localization system for operations in pipes such as those found in Liquified Natural Gas (LNG) production. A forward facing fisheye camera mounted on a prototype robot collects imagery as it is teleoperated through a pipe network. The images are processed offline to estimate camera pose and sparse scene structure where the results can be used to generate 3D renderings of the pipe surface. The method extends state of the art visual odometry and mapping for fisheye systems to incorporate geometric constraints based on prior knowledge of the pipe components into a Sparse Bundle Adjustment framework. These constraints significantly reduce inaccuracies resulting from the limited spatial resolution of the fisheye imagery, limited image texture, and visual aliasing. Preliminary results are presented for datasets collected in our fiberglass pipe network which demonstrate the validity of the approach.

Traditional lidar simulations render surface models to generate simulated range data. For objects with welldefined surfaces, this approach works well, and traditional 3D scene reconstruction algorithms can be employed to automatically generate the surface models. This approach breaks down, though, for many trees, tall grasses, and other objects with fine-scale geometry: surface models do not easily represent the geometry, and automated reconstruction from real data is difficult. In this paper, we introduce a new stochastic volumetric model that better captures the complexities of real lidar data of vegetation and is far better suited for automatic modeling of scenes from field collected lidar data. We also introduce several methods for automatic modeling and for simulating lidar data utilizing the new model. To measure the performance of the stochastic simulation we use histogram comparison metrics to quantify the differences between data produced by the real and simulated lidar. We evaluate our approach on a range of real world datasets and show improved fidelity for simulating geo-specific outdoor, vegetation scenes.

Pipe inspection is a critical process in many industries, including oil and gas. Conventional practice relies on a range of Non-Destructive Testing (NDT) approaches such as ultrasonic and magnetic flux leakage methods. While these approaches can provide high accuracy wall thickness measurements, which can be used to monitor corrosion, they provide poor visualizations, and are typically unable to provide full pipe coverage. Moreover, they cannot be used to localize where in the pipe a defect is without expensive and possibly restricted sensors such as Inertial Navigation Systems. We have developed an automated vision-based approach that builds high-resolution 3D appearance maps of pipes and provides vehicle localization. These maps include both structure and appearance information, and can be used for direct metric measurement of pipe wall thickness, or as input to automatic corrosion detection algorithms. They may also be imported into 3D rendering engines to provide effective visualization of a pipe network. Our most recent system uses a wide angle of view fisheye camera which enables full pipe coverage and is sufficiently compact for practical applications. Our approach to mapping and localization builds from state-of-the-art visual odometry methods and extends them to deal with (visually) feature poor engineered environments. We present the results of this work using image datasets collected within our constructed pipe network. A range of empirical results are presented to validate the approach.

Stereo vision is a common sensing technique for mobile robots and is becoming more broadly used in automotive, industrial, entertainment, and consumer products. The quality of range data from a stereo system is highly dependent on the intrinsic and extrinsic calibration of the sensor head. Unfortunately, for deployed systems, drift in extrinsic calibration is nearly unavoidable. Thermal variation and cycling combined with shock and vibration can cause transitory or permanent changes in extrinsics that are not modeled accurately by a static calibration. As a result the quality of the sensor degrades significantly. We have developed a new approach that provides real-time continuous calibration updates to extrinsic parameters. Our approach optimizes the extrinsic parameters to reduce epipolar errors over one or multiple frames. A Kalman Filter is used to continually refine these parameter estimates and minimize inaccuracies resulting from visual feature noise and spurious feature matches between the left and right images. The extrinsic parameter updates can be used to re-rectify the stereo imagery. Thus, it serves as a pre-processing step for any stereo process ranging, from dense reconstruction to visual odometry. We have validated our system in a range of environments and stereo tasks and demonstrated it at the recent Computer Vision and Pattern Recognition conference. Significant improvements to stereo visual odometry and scene mapping accuracy were achieved for datasets collected using both custom built and commercial stereo heads.

Stereo visual odometry and dense scene reconstruction depend critically on accurate calibration of the extrinsic (relative) stereo camera poses. We present an algorithm for continuous, online stereo extrinsic re-calibration operating only on sparse stereo correspondences on a per-frame basis. We obtain the 5 degree of freedom extrinsic pose for each frame, with a fixed baseline, making it possible to model time-dependent variations. The initial extrinsic estimates are found by minimizing epipolar errors, and are refined via a Kalman Filter (KF). Observation covariances are derived from the Cramer-Rao lower bound of the solution uncertainty. The algorithm operates at frame rate with unoptimized Matlab code with over 1000 correspondences per frame. We validate its performance using a variety of real stereo datasets and simulations.

High-fidelity simulation is a key enabling technology for the widespread deployment of large unmanned ground vehicles (UGVs). However, current approaches for lidar simulation leave much to be desired, particularly for scenes with vegetation. We introduce a novel 3D mapping technique that learns high-fidelity models for geo-specific lidar simulation directly from pose tagged lidar data. We introduce a novel stochastic, volumetric model that captures and can reproduce the statistical interactions of lidar with terrain. We show how to automatically learn the model directly from 3D mapping data collected by a UGV in the target environment. We extend our approach using terrain-classification techniques to develop a hybrid surface–volumetric model that combines the efficiency of surface modeling for areas that are well approximated by large surfaces (e.g. roads, bare earth) with our volumetric approach for more complex areas (e.g. bushes, trees) without sacrificing overall fidelity. We quantitatively compare the performance of our approach against more conventional methods on large outdoor datasets from urban and off-road environments. Our results show significant performance gains using our volumetric and hybrid approaches over the state-of-the-art, laying the ground work for truly high-fidelity simulation engines for UGVs.

Pipe inspection is a critical activity in gas production facilities and many other industries. In this paper, we contribute a stereo visual odometry system for creating high resolution, sub-millimeter maps of pipe surfaces. Such maps provide both 3D structure and appearance information that can be used for visualization, cross registration with other sensor data, inspection and corrosion detection tasks. We present a range of optical configuration and visual odometry techniques that we use to achieve high accuracy while minimizing specular reflections. We show empirical results from a range of datasets to demonstrate the performance of our approach.

Regular inspection for corrosion of the pipes used in Liquified Natural Gas (LNG) processing facilities is critical for safety. We argue that a visual perception system equipped on a pipe crawling robot can improve on existing techniques (Magnetic Flux Leakage, radiography, ultrasound) by producing high resolution registered appearance maps of the internal surface. To achieve this capability, it is necessary to estimate the pose of sensors as the robot traverses the pipes. We have explored two monocular visual odometry algorithms (dense and sparse) that can be used to estimate sensor pose. Both algorithms use a single easily made measurement of the scene structure to resolve the monocular scale ambiguity in their visual odometry estimates. We have obtained pose estimates using these algorithms with image sequences captured from cameras mounted on different robots as they moved through two pipes having diameters of 152mm (6") and 406mm (16"), and lengths of 6 and 4 meters respectively. Accurate pose estimates were obtained whose errors were consistently less than 1 percent for distance traveled down the pipe.

The task of teleoperating a robot over a wireless video link is known to be very difficult. Teleoperation becomes even more difficult when the robot is surrounded by dense obstacles, or speed requirements are high, or video quality is poor, or wireless links are subject to latency. Due to high-quality lidar data, and improvements in computing and video compression, virtualized reality has the capacity to dramatically improve teleoperation performance — even in high-speed situations that were formerly impossible. In this paper, we demonstrate the conversion of dense geometry and appearance data, generated on-the-move by a mobile robot, into a photorealistic rendering model that gives the user a synthetic exterior line-of-sight view of the robot, including the context of its surrounding terrain. This technique converts teleoperation into virtual line-of-sight remote control. The underlying metrically consistent environment model also introduces the capacity to remove latency and enhance video compression. Display quality is sufficiently high that the user experience is similar to a driving video game where the surfaces used are textured with live video.

Simulations provide a safe, controlled setting for testing and are therefore ideal for rapidly developing and testing autonomous mobile robot behaviors. However, algorithms for mobile robots are notorious for transitioning poorly from simulations to fielded platforms. The difficulty can in part be attributed to the use of simplistic sensor models that do not recreate important phenomena that affect autonomous navigation. The differences between the output of simple sensor models and true sensors are highlighted using results from a field test exercise with the National Robotics Engineering Center's Crusher vehicle. The Crusher was manually driven through an area consisting of a mix of small vegetation, rocks, and hay bales. LIDAR sensor data was collected along the path traveled and used to construct a model of the area. LIDAR data were simulated using a simple point-intersection model for a second, independent path. Cost maps were generated by the Crusher autonomy system using both the real-world and simulated sensor data. The comparison of these cost maps shows consistencies on most solid, large geometry surfaces such as the ground, but discrepancies around vegetation indicate that higher fidelity models are required to truly capture the complex interactions of the sensors with complex objects.

Qatar is a worldwide leader in liquid natural gas (LNG) production and is poised to lead the world in gas-to-liquids (GTL) production with the commissioning of the Pearl GTL facility. Unfortunately, Qatar's gas fields contain non-negligible quantities of corrosive and toxic hydrogen sulfide (H2S), resulting in the ongoing need for expensive and labor intensive pipe inspection to detect and monitor areas of corrosion. Such inspection is critical to plant integrity, worker safety, and to ensure the economic productivity of the facility. Current industry practice relies on manual sensors operated by a worker located externally to the pipe. The complex pipe geometries and sheer number of pipes, result in a sparse inspection process that forces inspectors to extrapolate measurements to large areas of the pipe network that are unseen. To overcome these limitations, we are pursuing a radically different approach that uses an articulated robot to navigate inside the pipe, combined with a vision-based perception system that can build a detailed, registered, high resolution 3D appearance map of the inside pipe surface. By using an articulated robot, we can significantly increase the direct measurement coverage of the pipe network. By using a vision-based perception system, we can build models for visualization of the inside pipe surface that can be directly evaluated for corrosion damage. Moreover, our approach lays the foundation for automating corrosion detection by enabling changes in co-registered multi-sensor fusion (e.g. using magnetic flux leakage) to be evaluated over time. Our work to date has focused on developing monocular and stereo visual odometry systems, which are the core component to building high resolution 3D appearance maps of the pipe surface from a robot crawler located inside the pipe. We have developed algorithms that take imagery collected from a robot moving inside the pipe, and are able to estimate the motion of the vehicle and the resulting structure and appearance of the pipe surface. We have evaluated our algorithms on pipe segments and have generated accurate, high resolution stitched images of the internal pipe surface. We will describe the details of our algorithms, current results, and next steps in our work.

This paper describes a method for creating photorealistic three-dimensional (3D) models of real-world environments in real-time for the purpose of improving and extending the capabilities of vehicle tele-operation. Our approach utilizes the combined data from a laser scanner (for modeling 3D geometry) and a video camera (for modeling surface appearance). The sensors are mounted on a moving vehicle platform, and a photo-realistic 3D model of the vehicle's environment is generated and displayed to the remote operator in real time. Our model consists of three main components: a textured ground surface, textured or colorized non-ground objects, and a textured background for representing regions beyond the laser scanner's sensing horizon. Our approach enables many unique capabilities for vehicle tele-operation, including viewing the scene from virtual viewpoints (e.g., behind the vehicle or top down), seamless augmentation of the environment with digital objects, and improved robustness to transmission latencies and data dropouts.