The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020 posed diverse challenges for unmanned aerial vehicles (UAVs). We present our four tailored UAVs, specifically developed for individual aerial-robot tasks of MBZIRC, including custom hardware- and software components. In Challenge 1, a target UAV is pursued using a high-efficiency, onboard object detection pipeline to capture a ball from the target UAV. A second UAV uses a similar detection method to find and pop balloons scattered throughout the arena. For Challenge 2, we demonstrate a larger UAV capable of autonomous aerial manipulation: Bricks are found and tracked from camera images. Subsequently, they are approached, picked, transported, and placed on a wall. Finally, in Challenge 3, our UAV autonomously finds fires using LiDAR and thermal cameras. It extinguishes the fires with an onboard fire extinguisher. While every robot features task-specific subsystems, all UAVs rely on a standard software stack developed for this particular and future competitions. We present our mostly open-source software solutions, including tools for system configuration, monitoring, robust wireless communication, high-level control, and agile trajectory generation. For solving the MBZIRC 2020 tasks, we advanced the state of the art in multiple research areas like machine vision and trajectory generation. We present our scientific contributions that constitute the foundation for our algorithms and systems and analyze the results from the MBZIRC competition 2020 in Abu Dhabi, where our systems reached second place in the Grand Challenge. Furthermore, we discuss lessons learned from our participation in this complex robotic challenge.

Every day, burning buildings threaten the lives of occupants and first responders trying to save them. Quick action is of essence, but some areas might not be accessible or too dangerous to enter. Robotic systems have become a promising addition to firefighting, but at this stage, they are mostly manually controlled, which is error-prone and requires specially trained personal. We present two systems for autonomous firefighting from air and ground we developed for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020. The systems use LiDAR for reliable localization within narrow, potentially GNSS-restricted environments while maneuvering close to obstacles. Measurements from LiDAR and thermal cameras are fused to track fires, while relative navigation ensures successful extinguishing. We analyze and discuss our successful participation during the MBZIRC 2020, present further experiments, and provide insights into our lessons learned from the competition.

Micro aerial vehicles (MAVs) have high potential for information gathering tasks to support situation awareness in search and rescue scenarios. Manually controlling MAVs in such scenarios requires experienced pilots and is error-prone, especially in stressful situations of real emergencies. The conditions of disaster scenarios are also challenging for autonomous MAV systems. The environment is usually not known in advance and GNSS might not always be available. We present a system for autonomous MAV flights in unknown environments which does not rely on global positioning systems. The method is evaluated in multiple search and rescue scenarios and allows for safe autonomous flights, even when transitioning between indoor and outdoor areas.

To meet the challenges involved in providing adequate robotic support to first responders, a holistic approach is needed. This requires close cooperation of first responders, researchers and companies for scenario-based needs analysis, iterative development of the corresponding system functionality and integrated robotic systems as well as human-robot teamwork support, and experimentation, system testing and evaluation in realistic missions carried out with or by first responders. We describe how such a holistic approach is implemented by the partners in the cooperative project A-DRZ for the establishment of the German Rescue Robotics Center (DRZ). The A-DRZ approach addresses important requirements identified by first responders: adaptation of operational capabilities of robotic platforms; robust network connectivity; autonomous assistance functions facilitating robot control; improving situation awareness for strategic and tactical mission planning; integration of human-robot teams in the first responders' mission command structure. Solutions resulting from these efforts are tested and evaluated in excercises utilizing the advanced capabilities at the DRZ Living Lab and in external deployments.

Micro aerial vehicles (MAVs), are frequently used for exploration, examination, and surveillance during search and rescue missions. Manually piloting these robots under stressful conditions provokes pilot errors and can result in crashes with disastrous consequences. Also, during fully autonomous flight, planned high-level trajectories can be erroneous and steer the robot into obstacles. In this work, we propose an approach to efficiently compute smooth, time-optimal trajectories MAVs that avoid obstacles. Our method first computes a trajectory from the start to an arbitrary target state, including position, velocity, and acceleration. It respects input- and state-constraints and is thus dynamically feasible. Afterward, we efficiently check the trajectory for collisions in the 3D-point cloud, recorded with the onboard lidar. We exploit the piecewise polynomial formulation of our trajectories to analytically compute axis-aligned bounding boxes (AABB) to speed up the collision checking process. If collisions occur, we generate a set of alternative trajectories in real-time. Alternative trajectories bring the MAV in a safe state, while still pursuing the original goal. Subsequently, we choose and execute the best collision-free alternative trajectory based on a distance metric. The evaluation in simulation and during a real firefighting exercise shows the capability of our method.

Visually guided control of micro aerial vehicles (MAV) demands for robust real-time perception, fast trajectory generation, and a capable flight platform. We present a fully autonomous MAV that is able to pop balloons, relying only on onboard sensing and computing. The system is evaluated with real robot experiments during the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020 where it showed its resilience and speed. In all three competition runs we were able to pop all five balloons in less than two minutes flight time with a single MAV.

Inspection of industrial chimneys and smoke pipes induces high costs due to production downtimes and imposes risks to the health of human workers due to high temperatures and toxic gases. We aim at speeding up and automating this process with multicopter micro aerial vehicles. To acquire high quality sensor data, flying close to the walls of the chimney is inevitable, imposing high demands on good localization and fast and reliable control. In this paper, we present an integrated chimney inspection system based on a small lightweight flying platform, well-suited for maneuvering in narrow space. For navigation and obstacle avoidance, it is equipped with a multimodal sensor setup including a lightweight rotating 3D laser scanner, stereo cameras for visual odometry and high-resolution surface inspection. We tested our system in a decommissioned industrial chimney at the Zollverein UNESCO world heritage site and present results from autonomous flights and reconstructions of the chimney surface.

In this work, we propose a method to efficiently compute smooth, time-optimal trajectories for micro aerial vehicles (MAVs) evading a moving obstacle. Our approach first computes an n-dimensional trajectory from the start- to an arbitrary target state including position, velocity and acceleration. It respects input- and state-constraints and is thus dynamically feasible. The trajectory is then efficiently checked for collisions, exploiting the piecewise polynomial formulation. If collisions occur, viastates are inserted into the trajectory to circumvent the obstacle and still maintain time-optimality. These viastates are described by position, velocity, and acceleration. The evaluation shows that the computational demands of the proposed method are minimal such that obstacle avoidance can begin within few milliseconds. Optimality of generated trajectories, combined with the ability for frequent online re-planning from non-hover initial conditions, make the approach well suited for evasion of suddenly perceived obstacles during fast flight.

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 has defined ambitious new benchmarks to advance the state-of-the-art in autonomous operation of ground-based and flying robots. This article covers our approaches to solve the two challenges that involved micro aerial vehicles (MAV). Challenge 1 required reliable target perception, fast trajectory planning, and stable control of an MAV in order to land on a moving vehicle. Challenge 3 demanded a team of MAVs to perform a search and transportation task, coined "Treasure Hunt", which required mission planning and multi-robot coordination as well as adaptive control to account for the additional object weight. We describe our base MAV setup and the challenge-specific extensions, cover the camera-based perception, explain control and trajectory-planning in detail, and elaborate on mission planning and team coordination. We evaluated our systems in simulation as well as with real-robot experiments during the competition in Abu Dhabi. With our system, we-as part of the larger team NimbRo-won the MBZIRC Grand Challenge and achieved a third place in both subchallenges involving flying robots.

Robots that solve complex tasks in environments too dangerous for humans to enter are desperately needed, e.g. for search and rescue applications. We describe our mobile manipulation robot Momaro, with which we participated successfully in the DARPA Robotics Challenge. It features a unique locomotion design with four legs ending in steerable wheels, which allows it both to drive omnidirectionally and to step over obstacles or climb. Furthermore, we present advanced communication and teleoperation approaches, which include immersive 3D visualization, and 6D tracking of operator head and arm motions. The proposed system is evaluated in the DARPA Robotics Challenge, the DLR SpaceBot Camp 2015, and lab experiments. We also discuss the lessons learned from the competitions and present initial steps towards autonomous operator assistance functions.

The past years have shown a remarkable growth in use-cases for micro aerial vehicles (MAVs). Conceivable indoor applications require highly robust environment perception, fast reaction to changing situations, and stable navigation, but reliable sources of absolute positioning like GNSS or compass measurements are unavailable during indoor flights. We present a high-performance autonomous inventory MAV for operation inside warehouses. The MAV navigates along warehouse aisles and detects the placed stock in the shelves alongside its path with a multimodal sensor setup containing an RFID reader and two high-resolution cameras. We describe in detail the SLAM pipeline based on a 3D lidar, the setup for stock recognition, the mission planning and trajectory generation, as well as a low-level routine for avoidance of dynamical or previously unobserved obstacles. Experiments were performed in an operative warehouse of a logistics provider, in which an external warehouse management system provided the MAV with high-level inspection missions that are executed fully autonomously.

Micro aerial vehicles (MAVs), such as multirotors, are envisioned for autonomous inventory-taking in large warehouses. Fully autonomous operation of MAVs in such complex 3D environments requires real-time state estimation, obstacle detection, mapping, and navigation planning. To this end, we employ a cognitive MAV equipped with multiple sensors including a dual 3D laser scanner, three stereo camera pairs, an IMU, an RFID reader, and a powerful onboard computer running the ROS middleware. Tasks with hard real-time requirements such as attitude control and state estimation are processed on a Pixhawk Autopilot, which communicates with the main computer via the MAVLink protocol. In this chapter, we describe our integrated system for autonomous MAV-based inventory in warehouses. We detail the involved components and evaluate our system with the real autonomous MAV in a realistic scenario. We also report lessons learned during field testing.

Micro aerial vehicles, such as multirotors, are developed for applications like autonomous monitoring, inspection, and surveillance. Most of the current application scenarios assume a stationary environment and thus, trajectory generation relies on static targets. In this paper, we address time-optimal trajectory generation in dynamic environments, e.g., for landing on a moving platform. We extend our existing trajectory generation method by the ability to synchronize several axes of motion, following a master/slave approach for individual axes. We further explicitly treat moving targets, like moving landing platforms by planning in a target-centric frame. Our evaluation demonstrates the performance of the method under disturbances. These results have application in dynamic multicopter flight, and also allow for fast and precise multicopter motion under challenging conditions.

The ability to identify, follow, approach, and intercept a non-stationary target is a desirable capability of autonomous micro aerial vehicles (MAV) and puts high demands on reliable target perception, fast trajectory planning, and stable control. We present a fully autonomous MAV that lands on a planar platform mounted on a ground vehicle, relying only on onboard sensing and computing. We evaluate our system in simulation as well as with real robot experiments. Its resilience was demonstrated at the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) where it worked under competition conditions. Our team NimbRo ranked third in the MBZIRC Challenge 1 and — in combination with two other tasks — won the MBZIRC Grand Challenge.

Picking and transporting objects in an outdoor environment with multiple lightweight MAVs is a demanding task. The main challenges are sudden changes of flight dynamics due to altered center of mass and weight, varying lighting conditions for visual perception, and coordination of the MAVs over unreliable wireless connections. At the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) teams competed in a Treasure Hunt where three MAVs had to collaboratively pick colored disks and drop them into a designated box. Only little preparation and test time on- site required robust algorithms and easily maintainable systems to successfully achieve the challenge objectives. We describe our multi-robot system employed at MBZIRC, including a lightweight gripper, a vision system robust against illumination and color changes, and a control architecture allowing to operate multiple robots safely. With our system, we-as part of the larger team NimbRo of ground and flying robots-won the Grand Challenge and achieved a third place in the Treasure Hunt.

Inspection of industrial chimneys and smoke pipes induces high costs due to production downtimes and imposes risks to the health of human workers due to high temperatures and toxic gases. We aim at speeding up and automating this process with sensors mounted on multicopters. To acquire high quality sensor data, flying close to the walls of the chimney is inevitable, imposing high demands on good localization and fast and reliable control. In this paper, we present an integrated chimney inspection robot based on a small lightweight flying platform, well-suited for maneuvering in narrow space. For navigation and obstacle avoidance, it is equipped with a multimodal sensor setup including a lightweight 3D laser scanner, stereo cameras, and a high-resolution camera for surface inspection. We tested our system in a mock-up chimney modeling several surfaces found in real chimneys, and present results from autonomous flights and the reconstruction of the chimney surface.

Robots that solve complex tasks in environments too dangerous for humans to enter are desperately needed, e.g. for search and rescue applications. We describe our mobile manipulation robot Momaro, with which we participated successfully in the DARPA Robotics Challenge. It features a unique locomotion design with four legs ending in steerable wheels, which allows it both to drive omnidirectionally and to step over obstacles or climb. Furthermore, we present advanced communication and teleoperation approaches, which include immersive 3D visualization, and 6D tracking of operator head and arm motions. The proposed system is evaluated in the DARPA Robotics Challenge, the DLR SpaceBot Cup Qualification and lab experiments. We also discuss the lessons learned from the competitions.