Inspection of underwater structures with tethered underwater vehicles is often hindered by the risk of tether entanglement. We propose REACT (real-time entanglement- aware coverage path planning for tethered underwater ve- hicles), a framework designed to overcome this limitation. REACT comprises a computationally efficient geometry-based tether model using the signed distance field (SDF) map for accurate, real-time simulation of taut tether configurations around arbitrary structures in 3D. This model enables an efficient online replanning strategy by enforcing a maximum tether length constraint, thereby actively preventing entanglement. By integrating REACT into a coverage path planning framework, we achieve safe and entanglement-free inspection paths, previously challenging due to tether constraints. The complete REACT framework’s efficacy is validated in a pipe inspection scenario, demonstrating safe navigation and full coverage inspection. Simulation results show that REACT achieves complete coverage while maintaining tether constraints and completing the total mission 20% faster than conventional planners, despite a longer inspection time due to proactive avoidance of entanglement that eliminates extensive post-mission disentanglement. Real-world experiments confirm these benefits, where REACT completes the full mission, while the baseline planner fails due to physical tether entanglement.

INDI-QP extends Incremental Nonlinear Dynamic Inversion (INDI) with a quadratic program to prioritize critical degrees of freedom during actuator failure. While the algorithm has gained significant attention within the aerial vehicle community, it has not yet been studied for underwater systems. In this work, we adapt INDI-QP for the 6-DOF autonomous underwater vehicle “Cuttlefish”, which is equipped with eight thrusters and evaluate it on a 360$^{\circ }$ inspection trajectory. To generate the inspection motion, a circular motion controller maintains a constant radius and attitude relative to a fixed object while keeping the line of sight aligned with the sensor axis. Our experiments show that, by prioritizing critical degrees of freedom, the proposed INDI-QP controller safely executes the inspection trajectory and maintains a significantly smaller line-of-sight off-axis error than a non-prioritized INDI baseline, even with fewer than six functional thrusters, at the cost of increased positional tracking error. Passive fault-tolerant control is enabled without thruster speed measurements by integrating a parallel thruster model into the control architecture.

This work proposes a method for learning features from a batch of 2D sonar images to predict a multi-view point-cloud for achieving a dense 3D-reconstruction. In comparison to vision-based sensors, acoustics are considered a reliable sensing modality in underwater environments. The output of sonars is a 2D image which is unable to represent the scanned scene in all three dimensions. Estimation of this missing information, known as the elevation angle, is the key to performing 3d-reconstruction from acoustic images. One of the approaches is to predict a depth-map from the 2D sonar image, and transforming it into a point-cloud. In this paper, this idea is further improved into learning features from a batch of 2D acoustic images and predicting multiple depthmaps of the scanned object which covers it from different viewpoints. For training the deep learning model, and due to the lack of datasets from real environments, data was generated synthetically. For reducing the simulation-to-real gap, a Cycle-GAN was trained on real images for transferring the realistic style into the synthetically generated images. The conducted experiments in simulation showed that the proposed method is able to perform dense 3D reconstruction. The approach was then further tested in a real environment using an underwater vehicle, which accurately 3d-reconstructed the scanned objects achieving an average chamfer distance error of 0.06 meters when compared to a laser-scanned ground-truth.

With over 80% of the world's oceans remaining unmapped and unexplored, the advancement of robust underwater perception technologies is becoming more important than ever before. This vast frontier cannot be reliably observed with optical cameras mounted on Autonomous Underwater Vehicles (AUVs), which struggle in turbid, low-light conditions common to marine environments. Acoustic sensors like the Forward-Looking Sonar (FLS) are essential alternatives, yet progress in the field is significantly hampered by a profound scarcity of large-scale, publicly available sonar datasets. To address this critical gap, we introduce the SonarCloud Dataset, a comprehensive synthetic dataset generated to accelerate research in underwater perception. Our dataset consists of FLS and depth imagery of 19 distinct objects, totaling approximately 500,000 images, along with a 3D point cloud for each object generated from the corresponding depth maps in various orientations. As technical validation, we selected object detection and 3D reconstruction to evaluate the effectiveness of our dataset. We demonstrate that state-of-theart models trained solely on simulated data from our dataset can successfully detect objects in real-world sonar images and reconstruct their 3D shapes. The SonarCloud Dataset is presented as a valuable tool for the research community, and it can be found publicly in: https://doi.org/10.5281/zenodo.16645568.

Exploration of the underwater domain has always been a challenging task for researchers to tackle. Perception being a major part of this exploration, requires robust systems and sensors to perform accurate mapping, detection, tracking, and 3d-reconstruction of the underwater medium. Although many use optical sensors for such tasks, they get highly affected by the reduced visibility and high turbidity causing the loss of essential features in captured scenes. This work proposes the use of a deep-learning model for acoustic-based 3D object detection. The idea is to detect regions of interest/objects in sparse multibeam echo sounder 3D maps which are low in resolution and consist of minimum amount of features. Since acquiring data from a real environment is hard, the model was trained on a synthetically generated datasets consisting of several objects. Simulation experiments showed promising results performing successful 3D object detection, which is then further tested in a real experiment on seven different objects. The system performed accurate 3D detection achieving an mAP of 0.75.

Sonar sensing is fundamental for underwater robotics, but limited by capabilities of AI systems, which need large training datasets. Public data in sonar modalities is lacking. This paper presents the Marine Debris Forward-Looking Sonar datasets, with three different settings (watertank, turntable, flooded quarry) increasing dataset diversity and multiple computer vision tasks: object classification, object detection, semantic segmentation, patch matching, and unsupervised learning. We provide full dataset description, basic analysis and initial results for some tasks. We expect the research community will benefit from this dataset, which is publicly available at https://doi.org/10.5281/zenodo.15101686

In this paper, we present a dataset related to the behaviour of underwater umbilicals of a Remotely Operated Vehicle (ROV), specifically the BlueROV2. The data were collected from three different sources: a Motion Capture System (mo-cap system), the onboard sensors of the BlueROV2, and a tension sensor. The mo-cap system tracks the motion of the ROV and its tether, while the tension sensor measures the force exerted by the cable tension on the surface side, near the tether drum. The dataset were acquired for research purposes, specifically for studying underwater cables in the context of modeling, estimation, and control using machine learning and data-driven control methods. They can serve as a benchmark for developing and validating underwater tether models, enabling researchers to develop control methods that consider tether drag or entanglement, improving navigation accuracy of ROVs, or developing automated tether management systems. This paper describes the experimental setup, data acquisition, and post-processing procedures. Furthermore, it provides illustrative plots to highlight key features of the dataset.

Machine learning proves effective in constructing dynamics models from data, especially for underwater vehicles. Continuous refinement of these models using incoming data streams, however, often requires storage of an overwhelming amount of redundant data. This work investigates the use of uncertainty in the selection of data points to rehearse in online learning when storage capacity is constrained. The models are learned using an ensemble of multilayer perceptrons as they perform well at predicting epistemic uncertainty. We present three novel approaches: the Threshold method, which excludes samples with uncertainty below a specified threshold, the Greedy method, designed to maximize uncertainty among the stored points, and Threshold-Greedy, which combines the previous two approaches. The methods are assessed on data collected by an underwater vehicle Dagon. Comparison with baselines reveals that the Threshold exhibits enhanced stability throughout the learning process and also yields a model with the least cumulative testing loss. We also conducted detailed analyses on the impact of model parameters and storage size on the performance of the models, as well as a comparison of three different uncertainty estimation methods.

Visual odometry (VO) relies on sequential camera images to estimate robot motion. For underwater robots, this is often complicated by turbidity, light attenuation, and environments containing scarce or repetitive features. Even ideal imagery suffers from the issue of scale ambiguity common to all monocular VO implementations. To address these issues, we supplement a camera with a multibeam echosounder. This acoustic, time-of-flight sensor comes with its own challenges, including relatively slow and sparse measurements that can be further degraded by backscatter from suspended particulate matter as well as interfering sounds from nearby marine traffic. We propose a method for fusing only data from these two inspection sensors into a hybrid VO solution that does not rely on IMU, DVL, or any other positioning sensor. We demonstrate this method on real data collected by an autonomous underwater vehicle performing end-to-end pipeline inspection in the open ocean, where multiple passes through the same scene (i.e., the "loop closure" common to SLAM algorithms) is often time and cost prohibitive. We also show how this approach can be extended for the creation of dense point clouds that provide a colored reconstruction of the surveyed scene.

In this paper, we propose a path-following controller for an autonomous surface vehicle (ASV) that is based on model-free deep reinforcement learning. To make the learning agent more robust, we investigate domain randomization for sim-to-real transfer. We provide a comparison between three different algorithms: Deep Deterministic Policy Gradient (DDPG), Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO). The trained models are evaluated on the small-scale ASV Altus-LSA Niriis in the maritime test basin at DFKI RIC, Germany. Our results show that applying domain randomization leads to a significant performance improvement compared to no domain randomization, when tested on real hardware. Copyright (C) 2024 The Authors. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/)

To explore oceans on ice-covered moons in the solar system, energy-efficient Autonomous Underwater Vehicles (AUVs) with long ranges must cover enough distance to record and collect enough data. These usually underactuated vehicles are hard to control when performing tasks such as vertical docking or the inspection of vertical walls. This paper introduces a control strategy for DeepLeng to navigate in the ice-covered ocean of Jupiter's moon Europa and presents simulation results preceding a discussion on what is further needed for robust control during the mission.

Autonomous docking for underwater vehicles, especially locating the docking station, presents significant challenges for deploying sub-sea resident AUVs in exploration and monitoring tasks. To extend a fiducial marker-based docking station detection we propose to use the state-of-the-art object detection deep learning models, specifically YOLOv8 in various sizes. We assess the robustness of these models in detecting docking stations by training different model sizes under various configurations on a dataset collected at the DFKI test basin in Bremen, Germany. To show and improve their performances in a real-world under-ice scenario we utilize a previously recorded dataset from Torneträsk lake in Abisko, Sweden [1]. The performance of these models is then compared to the previously used fiducial marker-based docking station detection. Our results show that a combination of both the classical detection method with one of the trained YOLOv8 models improves the detection performance significantly.

Metadata of a Large Sonar and Stereo Camera Dataset Suitable for Sonar-to-RGB Image Translation Introduction This is a set of metadata describing a large dataset of synchronized sonar and stereo camera recordings, that were captured between August 2021 and September 2023 during the project DeeperSense (https://robotik.dfki-bremen.de/en/research/projects/deepersense/), as training data for Sonar-to-RGB image translation. Parts of the sensor data have been published (https://zenodo.org/records/7728089, https://zenodo.org/records/10220989). Due to the size of the sensor data corpus, it is currently impractical to make the entire corpus accessible online. Instead, this metadatabase serves as a relatively compact representation, allowing interested researchers to inspect the data, and select relevant portions for their particular use case, which will be made available on demand. This is an effort to comply with the FAIR principle A2 (https://www.go-fair.org/fair-principles/) that metadata shall be accessible, even when the base data is not immediately. Locations and sensors The sensor data was captured at four different locations, including one laboratory (Maritime Exploration Hall at DFKI RIC Bremen) and three field locations (Chalk Lake Hemmoor, Tank Wash Basin Neu-Ulm, Lake Starnberg). At all locations, a ZED camera and a Blueprint Oculus M1200d sonar were used. Additionally, a SeaVision camera was used at the Maritime Exploration Hall at DFKI RIC Bremen and at the Chalk Lake Hemmoor. The examples/ directory holds a typical output image for each sensor at each available location. Data volume per session Six data collection sessions were conducted. The table below presents an overview of the amount of data captured in each session: Session dates Location Number of datasets Total duration of datasets [h] Total logfile size [GB] Number of images Total image size [GB] 2021-08-09 - 2021-08-12 Maritime Exploration Hall at DFKI RIC Bremen 52 10.8 28.8 389’047 88.1 2022-02-07 - 2022-02-08 Maritime Exploration Hall at DFKI RIC Bremen 35 4.4 54.1 629’626 62.3 2022-04-26 - 2022-04-28 Chalk Lake Hemmoor 52 8.1 133.6 1’114’281 97.8 2022-06-28 - 2022-06-29 Tank Wash Basin Neu-Ulm 42 6.7 144.2 824’969 26.9 2023-04-26 - 2023-04-27 Maritime Exploration Hall at DFKI RIC Bremen 55 7.4 141.9 739’613 9.6 2023-09-01 - 2023-09-02 Lake Starnberg 19 2.9 40.1 217’385 2.3 255 40.3 542.7 3’914’921 287.0 Data and metadata structure Sensor data corpus The sensor data corpus comprises two processing stages: raw data streams stored in ROS bagfiles (aka logfiles), camera and sonar images (aka datafiles) extracted from the logfiles. The files are stored in a file tree hierarchy which groups them by session, dataset, and modality: ${session_key}/ ${dataset_key}/ ${logfile_name} ${modality_key}/ ${datafile_name} A typical logfile path has this form: 2023-09_starnberg_lake/ 2023-09-02-15-06_hydraulic_drill/ stereo_camera-zed-2023-09-02-15-06-07.bag A typical datafile path has this form: 2023-09_starnberg_lake/ 2023-09-02-15-06_hydraulic_drill/ zed_right/ 1693660038_368077993.jpg All directory and file names, and their particles, are designed to serve as identifiers in the metadatabase. Their formatting, as well as the definitions of all terms, are documented in the file entities.json. Metadatabase The metadatabase is provided in two equivalent forms: as a standalone SQLite (https://www.sqlite.org/index.html) database file metadata.sqlite for users familiar with SQLite, as a collection of CSV files in the csv/ directory for users who prefer other tools. The database file has been generated from the CSV files, so each database table holds the same information as the corresponding CSV file. In addition, the metadatabase contains a series of convenience views that facilitate access to certain aggregate information. An entity relationship diagram of the metadatabase tables is stored in the file entity_relationship_diagram.png. Each entity, its attributes, and relations are documented in detail in the file entities.json Some general design remarks: For convenience, timestamps are always given in both a human-readable form (ISO 8601 formatted datetime strings with explicit local time zone), and as seconds since the UNIX epoch. In practice, each logfile always contains a single stream, and each stream is stored always in a single logfile. Per database schema however, the entities stream and logfile are modeled separately, with a “many-streams-to-one-logfile” relationship. This design was chosen to be compatible with, and open for, data collections where a single logfile contains multiple streams. A modality is not an attribute of a sensor alone, but of a datafile: Because a sensor is an attribute of a stream, and a single stream may be the source of multiple modalities (e.g. RGB vs. grayscale images from the same camera, or cartesian vs. polar projection of the same sonar output). Conversely, the same modality may originate from different sensors. As a usage example, the data volume per session which is tabulated at the top of this document, can be extracted from the metadatabase with the following SQL query: SELECT PRINTF( '%s - %s', SUBSTR(session_start, 1, 10), SUBSTR(session_end, 1, 10)) AS 'Session dates', location_name_english AS Location, number_of_datasets AS 'Number of datasets', total_duration_of_datasets_h AS 'Total duration of datasets [h]', total_logfile_size_gb AS 'Total logfile size [GB]', number_of_images AS 'Number of images', total_image_size_gb AS 'Total image size [GB]' FROM location JOIN session USING (location_id) JOIN ( SELECT session_id, COUNT(dataset_id) AS number_of_datasets, ROUND( SUM(dataset_duration) / 3600, 1) AS total_duration_of_datasets_h, ROUND( SUM(total_logfile_size) / 10e9, 1) AS total_logfile_size_gb FROM location JOIN session USING (location_id) JOIN dataset USING (session_id) JOIN view__dataset_total_logfile_size USING (dataset_id) GROUP BY session_id ) USING (session_id) JOIN ( SELECT session_id, COUNT(datafile_id) AS number_of_images, ROUND(SUM(datafile_size) / 10e9, 1) AS total_image_size_gb FROM session JOIN dataset USING (session_id) JOIN stream USING (dataset_id) JOIN datafile USING (stream_id) GROUP BY session_id ) USING (session_id) ORDER BY session_id;

Autonomous underwater docking is a necessary requirement for achieving long-term term residency for Autonomous Underwater Vehicles (AUVs). In this work, we propose a docking method that is capable of achieving robust docking by proposing a state machine and extending our localization framework with visual features from Apriltag marker measurements and USBL 2D position measurements. In total we executed 201 docking sequences of which 194 were succesful, thus yielding a total success rate of 96.5%. To further test the robustness of the proposed method, multiple experiments were conducted where feedback from visual markers was actively denied during the approach of the vehicle to its designated docking station. Our results show that the proposed method was able to complete 73 successful docking maneuvers out of 76 attempts (96%) without any visual aiding.

This work proposes the use of conditional Generative Adversarial Networks (cGANs) for acoustic-based 3D reconstruction. Acoustics being the most reliable sensor modality in underwater domains is accompanied with the loss of elevation angle in its images. The challenge of recovering the missing dimension in acoustic images have pushed researchers to try various methods and approaches over the past years. cGANs being an image-to-image translation method makes it possible to learn a desired style, and transforms the data from one modality to another. This was applied here as a way of transforming an acoustic image into another form which contains the elevation characteristics, such as depth images. Depth images are hard to acquire underwater, thus data was generated synthetically and used for training and testing the deep learning model. As a way of performance enhancement, real data was collected for training a Cycle-GAN network in the aim of transferring the realistic style into the synthetically generated images. Simulation experiments were conducted to evaluate the system and find out the best experimental setup, which was then used to carry out the real experiment. The system performed dense 3D reconstruction of the scanned object and proved to be applicable in real environments.

In this work we present a novel method for reconstructing 3D surfaces using a multi-beam imaging sonar. We integrate the intensities measured by the sonar from different viewpoints for fixed cell positions in a 3D grid. For each cell we integrate a feature vector that holds the mean intensity for a discretized range of viewpoints. Based on the feature vectors and independent sparse range measurements that act as ground truth information, we train convolutional neural networks that allow us to predict the signed distance and direction to the nearest surface for each cell. The predicted signed distances can be projected into a truncated signed distance field (TSDF) along the predicted directions. Utilizing the marching cubes algorithm, a polygon mesh can be rendered from the TSDF. Our method allows a dense 3D reconstruction from a limited set of viewpoints and was evaluated on three real-world datasets.

Self-supervised learning has proved to be a powerful approach to learn image representations without the need of large labeled datasets.For underwater robotics, it is of great interest to design computer vision algorithms to improve perception capabilities such as sonar image classification.Due to the confidential nature of sonar imaging and the difficulty to interpret sonar images, it is challenging to create public large labeled sonar datasets to train supervised learning algorithms.In this work, we investigate the potential of three self-supervised learning methods (RotNet, Denoising Autoencoders, and Jigsaw) to learn high-quality sonar image representation without the need of human labels.We present pre-training and transfer learning results on real-life sonar image datasets.Our results indicate that self-supervised pre-training yields classification performance comparable to supervised pre-training in a few-shot transfer learning setup across all three methods.Code and self-supervised pre-trained models are be available at agrija9/ssl-sonar-images.

Navigation data of the AUV DeepLeng during field trials in an ice-covered lake in Abisko, Sweden. For more information on this field trials: https://www.dfki.de/web/forschung/eurex-abisko This dataset is described in the paper M. Hildebrandt, T. Creutz, B. Wehbe, M. Wirtz and M. Zipper, "Under-Ice Field tests with an AUV in Abisko/Torneträsk," OCEANS 2022, Hampton Roads, Hampton Roads, VA, USA, 2022, pp. 1-7, doi: 10.1109/OCEANS47191.2022.9977094. For more details on how to use the data files, please refer to the Readme.md Contact: tom.creutz@dfki.de, bilal.wehbe@dfki.deFunded by BMWi (Kennziffer 50 NA 2002)

In this work we describe the field-tests with an AUV in an ice-covered lake in Abisko, Sweden in March 2022. The aim of the test campaign was to determine the vehicle's ability to operate under such conditions (temperature, limited accessibility) and perform a number of missions including autonomous docking to a docking-station. In order to aid other researchers in planning similar field-trials, we expand on the choice of venue and test-site set-up before presenting the results of multiple autonomous missions. Within this paper we provide the data of three missions including sensor readings of IMU, DVL, CTD and USBL as well as additional data and configuration parameters.