In this article, a novel approach for merging 3D point cloud maps in the context of egocentric multi-robot exploration is presented. Unlike traditional methods, the proposed approach leverages state-of-the-art place recognition and learned descriptors to efficiently detect overlap between maps, eliminating the need for the time-consuming global feature extraction and feature matching process. The estimated overlapping regions are used to calculate a homogeneous rigid transform, which serves as an initial condition for the GICP point cloud registration algorithm to refine the alignment between the maps. The advantages of this approach include faster processing time, improved accuracy, and increased robustness in challenging environments. Furthermore, the effectiveness of the proposed framework is successfully demonstrated through multiple field missions of robot exploration in a variety of different underground environments.

This paper addresses the problem of autonomous robotic inspection in complex and unknown environments. This capability is crucial for efficient and precise inspections in various real-world scenarios, even when faced with perceptual uncertainty and lack of prior knowledge of the environment. Existing methods for real-world autonomous inspections typically rely on predefined targets and waypoints and often fail to adapt to dynamic or unknown settings. In this paper, we introduce the Semantic Belief Behavior Graph (SB2G) framework as a new approach to semantic-aware autonomous robot inspection. SB2G generates a control policy for the robot, using behavior nodes that encapsulate various semantic-based policies designed for inspecting different classes of objects. We design an active semantic search behavior to guide the robot in locating objects for inspection while reducing semantic information uncertainty. The edges in the SB2G encode transitions between these behaviors. We validate our approach through simulation and real-world urban inspections using a legged robotic platform. Our results show that SB2G enables a more efficient object inspection policy, exhibiting similar behaviors comparable to human-operated inspections.

A staircase localization method is proposed for robots to explore urban environments autonomously. The proposed method employs a modular design in the form of a cascade pipeline consisting of three modules of stair detection, line segment detection, and stair localization modules. The stair detection module utilizes an object detection algorithm based on deep learning to generate a region of interest (ROI). From the ROI, line segment features are extracted using a deep line segment detection algorithm. The extracted line segments are used to localize a staircase in terms of position, orientation, and stair direction. The stair detection and localization are performed only with a single RGB-D camera. Each component of the proposed pipeline does not need to be designed particularly for staircases, which makes it easy to maintain the whole pipeline and replace each component with state-of-the-art deep learning detection techniques. The results of real-world experiments show that the proposed method can perform accurate stair detection and localization during autonomous exploration for various structured and unstructured upstairs and downstairs with shadows, dirt, and occlusions by artificial and natural objects.

Subterranean exploration and mapping for search and rescue robotics have become an emerging research direction since the DARPA organized Subterranean Challenge. As part of development efforts within the team CoSTAR (Collaborative SubTerranean Autonomous Robots) in the Sub-T challenge, this work establishes a novel traversable graph-based exploration strategy that utilizes frontiers for local navigation and a fast collision risk-aware graph building for global navigation. The exploration strategy extracts frontiers in an unknown area that contribute to safe navigation while maximizing information gain for the robot. The exploration problem is further bifurcated into local and global exploration for faster decision-making at junctions with the goal of rapidly exploring the area. The local exploration guarantees collision-free straight-line paths to informative frontiers for rapid forward navigation, while global re-positioning utilizes a traversable graph subject to geometrical collision checks within the occupancy map. The pathfinding in a graph is addressed using a heuristic, which combines risk margins and travel costs to assist in short yet safe paths to the global frontier in case of a dead end in local exploration. The presented exploration strategy is developed with the goal of making exploration algorithms platform agnostic in order to be able to use it with aerial, as well as ground robots. The proposed method is also evaluated against different state-of-the-art exploration planners in simulated fixed-time budget-based exploration missions on an Unmanned Aerial Vehicle (UAV) in order to benchmark the capabilities and highlight the novelty.

Exploration and mapping of unknown environments is a fundamental task in applications for autonomous robots. In this article, we present a complete framework for deploying Micro Aerial Vehicles (MAVs) in autonomous exploration missions in unknown subterranean areas. The main motive of exploration algorithms is to depict the next best frontier for the MAV such that new ground can be covered in a fast, safe yet efficient manner. The proposed framework uses a novel frontier selection method that also contributes to the safe navigation of autonomous MAVs in obstructed areas such as subterranean caves, mines, and urban areas. The framework presented in this work bifurcates the exploration problem in local and global exploration. The proposed exploration framework is also adaptable according to computational resources available onboard the MAV which means the trade-off between the speed of exploration and the quality of the map can be made. Such capability allows the proposed framework to be deployed in subterranean exploration and mapping as well as in fast search and rescue scenarios. The performance of the proposed framework is evaluated in detailed simulation studies with comparisons made against a high-level exploration-planning framework developed for the DARPA Sub-T challenge as it will be presented in this article.

To achieve autonomy in unknown and unstruc-tured environments, we propose a method for semantic-based planning under perceptual uncertainty. This capability is cru-cial for safe and efficient robot navigation in environment with mobility-stressing elements that require terrain-specific locomotion policies. We propose the Semantic Belief Graph (SBG), a geometric- and semantic-based representation of a robot's probabilistic roadmap in the environment. The SBG nodes comprise of the robot geometric state and the semantic-knowledge of the terrains in the environment. The SBG edges represent local semantic-based controllers that drive the robot between the nodes or invoke an information gathering action to reduce semantic belief uncertainty. We formulate a semantic-based planning problem on SBG that produces a policy for the robot to safely navigate to the target location with min-imal traversal time. We analyze our method in simulation and present real-world results with a legged robotic platform navigating multi-level outdoor environments.

Robust state estimation is critical for enabling reliable autonomous robot operations in challenging environments. To estimate the state, heterogeneous sensor fusion is commonly employed to enhance the reliability against perceptual failure. However, most known methods for sensor-fusion are brittle to dynamic perceptual condition changes due to the use of hand-tuned and time-constant error models. This paper introduces ROSE, a Robust Online-adaptive State Estimator, capable of adapting uncertainty statistics for individual multi-modal estimates in real-time to perform reliable sensor-fusion for robot state estimation. The proposed method leverages theory from adaptive Kalman filtering and extends it to optimization-based methods, to improve estimation accuracy while enabling integration of delayed heterogeneous sensor inputs. ROSE has been thoroughly evaluated by simulation studies and real-world demonstrations using a high-speed off-road vehicle navigating complex unstructured terrains and performing aggressive motions.

Robust mission planning is an essential component for mission autonomy to perform complicated tasks in extreme environments. In this paper, we are interested in the role of semantic abstractions for guiding autonomous mission planning. In particular, we focus on how semantics can be leveraged to transition, at the mission level, in between individually robust task plans. We present a mission autonomy framework wherein a task plan adaptation policy leverages up-to-date semantics information in order to adapt to changes that occur during run-time, which endows the robot with better resiliency to unexpected events and improves the overall efficiency of mission operations. Under this new perspective, we provide a concrete and challenging application of autonomous exploration and radio source seeking in a complex multi-level building environment. Experimental results over simulations and real hardware tests demonstrate that the presented semantics-aware mission adaptation more effectively completes the mission with better qualitative results compared to a non-adaptive baseline.

This article presents a 3D point cloud map-merging framework for egocentric heterogeneous multi-robot exploration, based on overlap detection and alignment, that is independent of a manual initial guess or prior knowledge of the robots' poses. The novel proposed solution utilizes state-of-the-art place recognition learned descriptors, that through the framework's main pipeline, offer a fast and robust region overlap estimation, hence eliminating the need for the time-consuming global feature extraction and feature matching process that is typically used in 3D map integration. The region overlap estimation provides a homogeneous rigid transform that is applied as an initial condition in the point cloud registration algorithm Fast-GICP, which provides the final and refined alignment. The efficacy of the proposed framework is experimentally evaluated based on multiple field multi-robot exploration missions in underground environments, where both ground and aerial robots are deployed, with different sensor configurations.

Modeling dynamics is often the first step to making a vehicle autonomous. While on-road autonomous vehicles have been extensively studied, off-road vehicles pose many challenging modeling problems. An off-road vehicle encounters highly complex and difficult-to-model terrain/vehicle interactions, as well as having complex vehicle dynamics of its own. These complexities can create challenges for effective high-speed control and planning. In this paper, we introduce a framework for multistep dynamics prediction that explicitly handles the accumulation of modeling error and remains scalable for sampling-based controllers. Our method uses a specially-initialized Long Short-Term Memory (LSTM) over a limited time horizon as the learned component in a hybrid model to predict the dynamics of a 4-person seating all-terrain vehicle (Polaris S4 1000 RZR) in two distinct environments. By only having the LSTM predict over a fixed time horizon, we negate the need for long term stability that is often a challenge when training recurrent neural networks. Our framework is flexible as it only requires odometry information for labels. Through extensive experimentation, we show that our method is able to predict millions of possible trajectories in real-time, with a time horizon of five seconds in challenging off road driving scenarios.

This article develops an impact-resilient aerial robot (s-ARQ) equipped with a compliant arm to sense contacts and reduce collision impact and featuring a real-time contact force estimator and a nonlinear motion controller to handle collisions while performing aggressive maneuvers and stabilize from high-speed wall collisions. Further, a new collision-inclusive planning method that aims to prioritize contacts to facilitate aerial robot navigation in cluttered environments is proposed. A range of simulated and physical experiments demonstrate key benefits of the robot and the contact-prioritized (CP) planner. Experimental results show that the compliant robot has only a 4% weight increase but around 40% impact reduction in drop tests and wall collision tests. s-ARQ can handle collisions while performing aggressive maneuvers and stabilize from high-speed wall collisions at 3.0 m/s with a success rate of 100%. Our proposed compliant robot and CP planning method can accelerate computation time while having shorter trajectory time and larger clearances compared to A* and RRT* planners with velocity constraints. Online planning tests in partially known environments further demonstrate the preliminary feasibility of our method to apply in practical use cases.

In this article, we propose a novel LiDAR and event camera fusion modality for subterranean (SubT) environments for fast and precise object and human detection in a wide variety of adverse lighting conditions, such as low or no light, high-contrast zones and in the presence of blinding light sources. In the proposed approach, information from the event camera and LiDAR are fused to localize a human or an object-of-interest in a robot's local frame. The local detection is then transformed into the inertial frame and used to set references for a Nonlinear Model Predictive Controller (NMPC) for reactive tracking of humans or objects in SubT environments. The proposed novel fusion uses intensity filtering and K-means clustering on the LiDAR point cloud and frequency filtering and connectivity clustering on the events induced in an event camera by the returning LiDAR beams. The centroids of the clusters in the event camera and LiDAR streams are then paired to localize reflective markers present on safety vests and signs in SubT environments. The efficacy of the proposed scheme has been experimentally validated in a real SubT environment (a mine) with a Pioneer 3AT mobile robot. The experimental results show real-time performance for human detection and the NMPC-based controller allows for reactive tracking of a human or object of interest, even in complete darkness.

In this paper, we present a LiDAR Inertial Odometry (LIO) algorithm utilizing adaptive keyframe generation which achieves fast and accurate state estimation for aerial and ground robots. It is known that keyframe generation significantly affects the performance of Simultaneous Localization and Mapping (SLAM) algorithms. Unlike existing SLAM algorithms that generate keyframes based on fixed conditions, we propose to use adaptive keyframe generation conditions considering characteristics of surrounding environment using real-time LiDAR scans. When a keyframe is generated, the keyframe and the corresponding LiDAR measurements are stored in our novel data structure designed for efficient sub- map generation. The scan to sub-map matching module then uses the Generalized Iterative Closest Point (GICP) algorithm to adjust estimated states at a global scale, producing more accurate and globally consistent state estimation results even in large-scale underground environments. Experimental results from diverse types of underground environments show that the proposed method outperforms the existing state-of-the-art LIO algorithms in various metrics such as computational speed, CPU usage, and accuracy.

Confined and subterranean areas are common in many civilian and industrial sites, although they are hazardous for humans given the presence of noxious gases, extreme temperatures, narrow spaces, unhealthy oxygen levels, flooding, and collapsing structures. Therefore, exploration, routine inspections, and surveillance tasks can benefit from using autonomous mobile robots to improve safety by reducing the presence of humans in those scenarios. However, despite advances in the field, there are still challenges to overcome for confined and subterranean robot operation. Real word robotic exploration requires robust and reliable map generation, precise localization, safe navigation, and efficient path planning. These requirements make exploration in complex 3D environments with rugged terrain difficult. The challenge is increased when considering multi-robot teams, as there is no guarantee of a functional network infrastructure. Despite consistent increasing interest in the area, there is a lack of research summarizing the results and best practices for exploring such environments. Therefore, in this paper, we provide a review and discuss state-of-the-art robotic exploration techniques, including single and cooperative approaches with homogeneous and heterogeneous teams, with a focus on complex subterranean and confined 3D scenarios. We also present a comprehensive list of insights on open challenges and possible directions for future investigation in the topic. • A survey in autonomous exploration for subterranean and confined spaces using mobile robots. • Description of current state of the art, including results from the DARPA subt Challenge. • Sensor classification for confined spaces inspection. • Comparison of recent works in autonomous exploration using multi-robot systems.

Mapping and exploration of a Martian terrain with an aerial vehicle has become an emerging research direction, since the successful flight demonstration of the Mars helicopter Ingenuity. Although the autonomy and navigation capability of the state of the art Mars helicopter has proven to be efficient in an open environment, the next area of interest for exploration on Mars are caves or ancient lava tube like environments, especially towards the never-ending search of life on other planets. This article presents an autonomous exploration mission based on a modified frontier approach along with a risk aware planning and integrated collision avoidance scheme with a special focus on energy aspects of a custom designed Mars Coaxial Quadrotor (MCQ) in a Martian simulated lava tube. One of the biggest novelties of the article stems from addressing the exploration capability, while rapidly exploring in local areas and intelligently global re-positioning of the MCQ when reaching dead ends in order to to efficiently use the battery based consumed energy, while increasing the volume of the exploration. The proposed three layer cost based global re-position point selection assists in rapidly redirecting the MCQ to previously partially seen areas that could lead to more unexplored part of the lava tube. The Martian fully simulated mission presented in this article takes into consideration the fidelity of physics of Mars condition in terms of thin atmosphere, low surface pressure and low gravity of the planet, while proves the efficiency of the proposed scheme in exploring an area that is particularly challenging due to the subterranean-like environment. The proposed exploration-planning framework is also validated in simulation by comparing it against the graph based exploration planner.

Experimental design of autonomous systems involves defining experimental inputs to maximize the experimenter’s information gained, minimize costs, and balance risk. This effectively leads to improved understanding and trustworthiness, which are necessary for deployment in realworld settings. Since experimental design is inherently a human-in-the-loop, sequential decision making problem, and decisions are being made about complex systems, an investigation into decision-making quality and decision-supporting methods is warranted. In this work, we investigate a decision support system (DSS) to augment the human’s experimental design decision making abilities, and conduct an exploratory user study to investigate the potential for decision support. Our findings show that experimenters, including experienced field roboticists, make suboptimal decisions and mistakes during the experimental design process, which suggests robotics research could benefit from DSSs. Our proposed DSS shows promise in some select aspects of experimental design, including helping to reduce suboptimal decisions, and participants in the user study reported favorable opinions of using such a system, including a sense of usefulness and lack of burden. The broader implication of this work is the identification of decision support in experimental design as one way to help bridge the gap between academia and industry by way of accelerated, informative experimentation and increased system explainability.

Autonomous driving is complex, requiring sophisticated 3D scene understanding, localization, mapping, and control. Rather than explicitly modelling and fusing each of these components, we instead consider an end-to-end approach via reinforcement learning (RL). However, collecting exploration driving data in the real world is impractical and dangerous. While training in simulation and deploying visual sim-to-real techniques has worked well for robot manipulation, deploying beyond controlled workspace viewpoints remains a challenge. In this paper, we address this challenge by presenting Sim2Seg, a re-imagining of RCAN that crosses the visual reality gap for off-road autonomous driving, without using any real-world data. This is done by learning to translate randomized simulation images into simulated segmentation and depth maps, subsequently enabling real-world images to also be translated. This allows us to train an end-to-end RL policy in simulation, and directly deploy in the real-world. Our approach, which can be trained in 48 hours on 1 GPU, can perform equally as well as a classical perception and control stack that took thousands of engineering hours over several months to build. We hope this work motivates future end-to-end autonomous driving research.

We present a method for autonomous exploration of large-scale unknown environments under mission time con-straints. We start by proposing the Frontloaded Information Gain Orienteering Problem (FIG-OP) - a generalization of the traditional orienteering problem where the assumption of a reliable environmental model no longer holds. The FIG-OP ad-dresses model uncertainty by frontloading expected information gain through the addition of a greedy incentive, effectively expe-diting the moment in which new area is uncovered. In order to reason across multi-kilometer environments, we solve FIG-OP over an information-efficient world representation, constructed through the aggregation of information from a topological and metric map. Our method was extensively tested and field-hardened across various complex environments, ranging from subway systems to mines. In comparative simulations, we observe that the FIG-OP solution exhibits improved coverage efficiency over solutions generated by greedy and traditional orienteering-based approaches (i.e. severe and minimal model uncertainty assumptions, respectively).

Search and rescue with a team of heterogeneous mobile robots in unknown and large-scale underground environments requires high-precision localization and mapping. This crucial requirement is faced with many challenges in complex and perceptually-degraded subterranean environments, as the onboard perception system is required to operate in off-nominal conditions (poor visibility due to darkness and dust, rugged and muddy terrain, and the presence of self-similar and ambiguous scenes). In a disaster response scenario and in the absence of prior information about the environment, robots must rely on noisy sensor data and perform Simultaneous Localization and Mapping (SLAM) to build a 3D map of the environment and localize themselves and potential survivors. To that end, this letter reports on a multi-robot SLAM system developed by team CoSTAR in the context of the DARPA Subterranean Challenge. We extend our previous work, LAMP, by incorporating a single-robot front-end interface that is adaptable to different odometry sources and lidar configurations, a scalable multi-robot front-end to support inter- and intra-robot loop closure detection for large scale environments and multi-robot teams, and a robust back-end equipped with an outlier-resilient pose graph optimization based on Graduated Non-Convexity. We provide a detailed ablation study on the multi-robot front-end and back-end, and assess the overall system performance in challenging real-world datasets collected across mines, power plants, and caves in the United States. We also release our multi-robot back-end datasets (and the corresponding ground truth), which can serve as challenging benchmarks for large-scale underground SLAM.

Lidar odometry has attracted considerable attention as a robust localization method for autonomous robots operating in complex GNSS-denied environments. However, achieving reliable and efficient performance on heterogeneous platforms in large-scale environments remains an open challenge due to the limitations of onboard computation and memory resources needed for autonomous operation. In this work, we present LOCUS 2.0, a robust and computationally-efficient lidar odometry system for real-time underground 3D mapping. LOCUS 2.0 includes a novel normals-based Generalized Iterative Closest Point (GICP) formulation that reduces the computation time of point cloud alignment, an adaptive voxel grid filter that maintains the desired computation load regardless of the environment’s geometry, and a sliding-window map approach that bounds the memory consumption. The proposed approach is shown to be suitable to be deployed on heterogeneous robotic platforms involved in large-scale explorations under severe computation and memory constraints. We demonstrate LOCUS 2.0, a key element of the CoSTAR team’s entry in the DARPA Subterranean Challenge, across various underground scenarios. We release LOCUS 2.0 as an open-source library and also release a lidar-based odometry dataset in challenging and large-scale underground environments. The dataset features legged and wheeled platforms in multiple environments including fog, dust, darkness, and geometrically degenerate surroundings with a total of $\text{11}~h$ of operations and $\text{16}~km$ of distance traveled.