Autonomous underwater vehicles (AUVs) equipped with side-looking sonars have become vital tools for seafloor exploration due to the combination of high image resolution and high area coverage rates. To reach their full operational performance AUVs also need onboard perception, including recognition of relevant objects. We combine adaptive template matching and real-time image simulation for automatic target recognition in synthetic aperture sonar images. We hypothesize that dynamic, rapid and fine-tuned search of object types and configurations should improve classification results and real-time responses. Analyses of experimental data with cylindrical objects outside of Horten, Norway, recorded by the Kongsberg Maritime HISAS1030 sonar, strengthened the hypothesis. Our setup outperformed a well-configured, static template database at false positive rates (FPR) above 10%-20%, with an area under curve improvement of one to two percent, depending on the correlation methods used. The system is implemented on a graphics processing unit using OpenGL and OpenCL, a computer graphics and general-purpose programming library, respectively. This facilitates a faster and more flexible classification process. We describe the implementation and provide a supplementary Python script to showcase the notation and implementation in practice.

Optical imaging for identifying targets of interest is an important operational phase in many applications of autonomous underwater vehicles (AUVs). However, underwater optical imaging is challenging due to limited visibility and rugged terrain. To counter the low visibility, the AUV needs to fly at a low altitude over the targets. This results in a small optical footprint and corresponding small margins for error in maneuvering and navigation accuracy, in addition to a significant risk of collision with the seabed. In this paper, we propose a planning algorithm that adapts the mission to in-situ knowledge for safe and robust optical inspection of seafloor objects. By using automatic target recognition on data from a multibeam echosounder with a wider field of view, the algorithm checks whether the target was within the camera footprint. For targets that were deemed outside, i.e., missed, the plan is updated with the corrected object positions. This significantly increases the probability of successful optical inspection in a reduced amount of time, as well as reduces the risk of collision. The proposed method has been demonstrated in sea experiments using a HUGIN AUV.

The process of reviewing seabed survey imagery for defence applications is time consuming for naval operators, and this workload is expected to increase with the large data volumes generated as high resolution synthetic aperture sonar (SAS) comes into service. Change detection is a means of reliably detecting newly occurring, moved, or removed seabed objects. It is particularly effective in cluttered environments such as harbours, or in areas that are routinely surveyed. While this technique offers a robust solution to the workload problem, in-service operator tools for post-mission analysis of survey imagery that are specifically designed for change detection are lacking. Coherent change detection (CCD), operating on complex-valued SAS imagery, offers the possibility of detecting very subtle seabed changes, and is an area of current research interest that is not yet implemented in operational tools. Automated change detection (ACD) processing performed on-board the sonar platform during the survey will provide the largest operational utility. Recognizing the complementary aspects of the three nations' naval research programs in this area, the US (NSWC-PCD, ARL Penn State), Canada (DRDC), and Norway (FFI) have formed a collaborative project to advance operator aids for change detection: Coalition Underwater Mine and IED Defeat (CUMID). Goals of the joint program are to: improve robustness of image-based ACD algorithms; draft requirements and specifications for ACD performance assessment, including both evaluation of past or ongoing performance and prediction of future performance; and architect operator displays, tools, and decision aids. The envisioned final output of the collaboration is a set of well-crafted requirements and recommendations that can be implemented in a manner fitting national priorities and capabilities. This paper provides an overview of current change detection practice (the state of the art), the coalition project, activities ongoing in the participating nations' research programs, highlights of workshop outputs, and plans for the future.

Classification of objects in synthetic aperture sonar (SAS) images is a vital task in underwater automatic target recognition (ATR) and deep learning has proven highly successful in this task. Typical deep learning systems used for processing of SAS images are inspired by results in the domain of optical images. However, unlike the common optical images, SAS images can be supplemented with additional meta-information such as the imaging geometry, spatial resolution and signal-to-noise ratio. This paper explores techniques to exploit imaging geometry as an additional source of information for improving the classification performance of SAS images with deep neural networks.

This paper presents fast, accurate, and automatic methods for detecting seafloor pipelines in multibeam echo sounder data with deep learning. The proposed methods take inspiration from the highly successful ResNet and YOLO deep learning models and tailor them to the idiosyncrasies of the seafloor pipeline detection task. We use the area between lines and Hausdorff line distance functions to accurately evaluate how well methods can localize (pipe)lines. The same functions also show promise as loss functions compared to standard mean squared error, which does not include the regression variables' geometrical interpretation. The model outperforms the highest likelihood baseline by more than 35% on a region-wise F1-score classification evaluation while being more than eight times more accurate than the baseline in locating pipelines. It is efficient, operating at over eighteen 32-ping image segments per second, which is far beyond real-time requirements.

This paper describes a novel technique for estimating how many mines remain after a full or partial underwater mine hunting operation. The technique applies Bayesian fusion of all evidence from the heterogeneous sensor systems used for detection, classification, and identification of mines. It relies on through-the-sensor (TTS) assessment, by which the sensors’ performances can be measured in situ through processing of their recorded data, yielding the local mine recognition probability, and false alarm rate. The method constructs a risk map of the minefield area composed of small grid cells (~4 m2) that are colour coded according to the remaining mine probability. The new approach can produce this map using the available evidence whenever decision support is needed during the mine hunting operation, e.g., for replanning purposes. What distinguishes the new technique from other recent TTS methods is its use of Bayesian networks that facilitate more complex reasoning within each grid cell. These networks thus allow for the incorporation of two types of evidence not previously considered in evaluation: the explosions that typically result from mine neutralization and verification of mine destruction by visual/sonar inspection. A simulation study illustrates how these additional pieces of evidence lead to the improved estimation of the number of deployed mines (M), compared to results from two recent TTS evaluation approaches that do not use them. Estimation performance was assessed using the mean squared error (MSE) in estimates of M.

This paper presents a novel method for automatically selecting the highest quality image sections for an underwater optical mosaic. We have constructed an image quality measure, which is used to determine the optimal seam between two overlapping images. We also present our preprocessing algorithm, which is a novel combination of existing methods. The full mosaic processing chain is demonstrated on images collected with a HUGIN autonomous underwater vehicle (AUV), using a camera system consisting of a high-sensitivity, still-image camera and a synchronized strobe with multiple light emitting diodes (LEDs).

Classification is an important operation in Automatic Target Recognition in sonar images. However, when a sonar image snippet contains an object belonging to a new class that was not seen during the training phase, the classifier cannot assign the correct class to it. Meta-learning is a solution to this problem. In one form of meta-learning, the original system is trained to emit a similarity measure between a support image and a query image. Then in inference, a given query image is classified into the class of the support image that has the highest similarity with the query image. Inference in such a system can be viewed as ”inverse image search”. In this work, we consider the problem of inverse image search in the context of an target recognition operator support module for synthetic aperture sonar images. The core challenge in building a system for image retrieval based on inverse image search is to design an algorithm that can output a relevant similarity measure for two input images. In this paper, we use a convolutional neural network (CNN) for similarity computation. Our experiments show that these measures are able to capture the similarity as perceived by humans and independent of class.

Synthetic aperture sonar (SAS) is emerging as the reference technique for high resolution imaging of the seabed. Given the insufficient positioning accuracy, the harsh ocean environment and lack of stability on relatively small platforms carrying the sonar, an optimal SAS image quality is not always guaranteed. In this paper, we present a technique to quantify the image quality based on the actual SAS images through-the-sensor. We have previously presented approaches to predict the sharpness in SAS images from meta-data, but in practice those have not always been precise given their predictive nature. We now introduce an image-based sharpness estimate that can be combined with the meta-data prediction. The image-based estimate is derived by locating candidate point scatterers and estimating key parameters from their point-spread functions. These provide information about defocus and unwanted grating lobes, the two most common types of image degradation in SAS. This technique requires texture in the image. For areas where no point scatterers can be detected, we rely only on meta-data to predict sharpness.

Many NATO navies are in the process of replacing their dedicated minehunting vessels with systems of heterogeneous, unmanned modules. While traditional ship-based assets prosecute sonar contacts in sequence through to neutralisation, modern systems employ unmanned vehicles equipped with side-looking sonar to detect and classify minelike contacts in a full area segment before proceeding with contact identification and mine neutralisation. This shift in technology and procedure brings important operational advantages, but also introduces a need to modify the traditional minehunting performance evaluation based on the percentage clearance metric. Previous works have demonstrated that the achieved detection and classification performance of modern minehunting systems can be estimated from the collected sonar data (through-the-sensor) and reported as detailed geographical maps. This paper extends the map-based evaluation approach to the identification and neutralisation phases, and also includes the case where some of the contacts or mines intentionally are left unprosecuted, e.g. disposal of only the specific mines required for establishing a safe sailing route. Each map cell is assumed to be sufficiently small to contain at most one sonar contact and can thus be assigned a status based on the hunting results for that cell: minelike contact, identified mine, etc. To this end we derive Bayesian formulations of a new performance metric: the probability of a remaining mine in a given cell. Furthermore, we show that this metric provides consistent multi-phase performance evaluation and estimates of the mine impact risk for a follow-on ship transiting a specified route.

FFI developed the tool “MCM Insite” for Mine Countermeasures (MCM) performance evaluation for Autonomous Underwater Vehicles (AUVs) equipped with side looking sonars. It estimates image quality and image complexity automatically from sonar images. The concept has been successfully applied to different synthetic aperture sonar (SAS) sensor systems taking full advantage of constant spatial resolution and phase information. However, it is more difficult to extract the required image metrics with non-interferometric, high-frequency sidescan sonar (SSS) systems, which are widely used for imaging the seafloor during minehunting operations. The main challenge of assessing image quality with SSS is to find a good signal-to-noise ratio (SNR) estimation from its common beamformed amplitude output. Also, for image complexity, the varying along-track resolution over range imposes a challenge when using scale-based image texture techniques. In this paper we aim for a solution for Edgetech SSS at 850 kHz frequency demonstrated on shallow water data.

In April 2016, NTNU, FFI, Kongsberg Seatex, LSTS and Maritime Robotics set up an experiment to explore their capability to combine the research vessel Gunnerus, the AUV Hugin, the UAV X8 and the USV Mariner in a network of heterogeneous unmanned vehicles. Communication, manoeuvring, onboard processing and operational complexity are essential components in such networks. To provide communication the MBR broadband radio system was implemented. To show the capabilities of the system proposed, a scenario with seabed mapping and target recognition was defined. The experiment made it apparent that these networks has the potential of significantly saving cost for data collection in marine research and management by reducing ship time. To fully unlock the potential of networks of heterogeneous unmanned vehicles, the missions of each vehicle need to be more integrated.

The adoption of autonomous underwater vehicles equipped with high-resolution side-looking sonars in mine countermeasures (MCM) operations has resulted in a requirement to quantify the effectiveness of these platforms; e.g., by the probability of correctly detecting and classifying mines along with the associated false alarm rate. Recently, the Norwegian Defence Research Establishment (FFI) proposed a performance model based on a combination of parameters given a priori and measured in situ, resulting in a through-the-sensor approach to performance evaluation for the detection and classification part of MCM operations. In this paper we investigate whether these parameters are consistent across different sonar systems and frequencies. This is done on basis of data from the MANEX'14 sea trial off the coast of Levanto, Italy, which was organized by the Centre for Maritime Research and Experimentation (CMRE). This unique data set spans five different frequency bands with carrier frequencies at 25 kHz, 72.5 kHz, 100 kHz, 300 kHz and 900 kHz, gathered from two synthetic aperture sonar systems (CMRE's MUSCLE and FFI's HISAS) as well as a real aperture sidescan sonar system.

This paper presents a concept and algorithms to detect, classify and identify mine-like objects within a single mission with an autonomous underwater vehicle. The autonomous mine hunting concept has been developed for the HUGIN series of vehicles. First, the operation area is surveyed either with a synthetic aperture sonar or a side-scanning sonar. During the survey, mine-like objects are detected and classified in the data using algorithms for automatic target recognition. When the survey is complete, a framework for autonomy initiates a fusion of the targets and starts the automatic planning of a mission plan for target identification. The autonomous mine hunting concept is a part of the development of a framework for advanced autonomy on HUGIN, the HUGIN autonomy layer. Implementation of this framework will reduce the risk of long-term AUV missions, and will provide intelligent vehicle behavior not only to re-inspect interesting objects and areas, but also to preserve vehicle safety, navigational accuracy and mission goals.

External inspection of seafloor pipelines is presently carried out with towed or remotely operated vehicles (ROV) operated from advanced offshore vessels, making it a time-consuming and complex activity. In co-operation with Kongsberg Maritime, FFI is developing a concept based on autonomous underwater vehicles (AUV) that has potential as a cost-effective augmentation of the ROV-based inspection. The concept was successfully demonstrated with a HUGIN 1000 AUV along a 30 km long section of an oil & gas pipeline on the western coast of Norway in February 2011. The AUV surveys the pipeline unaccompanied by the surface ship, which is then relieved to perform parallel inspection with ROV. The AUV must follow the pipeline within a cross-track range interval given by the sensor swaths. Preprogrammed mission paths may then be inadequate, in case of large prior uncertainties in pipe route and drift in vehicle position estimates. To ensure optimal pipe following, the pipeline should be automatically recognized in the sensor data and the vehicle path adjusted accordingly. This paper addresses the detection and tracking tasks when the AUV is travelling 50-100 meters to the side of the pipeline, imaging the pipeline and its surrounding seafloor with long-range, high resolution synthetic aperture sonar (SAS).

In this paper we present a planning algorithm for identification missions in an autonomous mine countermeasure scenario using AUVs. The concept of autonomous MCM identification missions is to first perform an autonomous survey of the mission area, during which an automatic target recognition (ATR) algorithm is run on the collected side-scan data. The ATR software detects possible contacts in the data, and classifies them as either clutter or a selection of mine classes. When the detection and classification are completed, the contact list is processed. In this processing contact with low detection confidence and classification confidence are discarded, while the rest are sent through a fusion process. In this process close contacts are fused together and their position averaged. After the contacts have been processed the list is sent to the identification planner. The identification planner will then create a mission plan with the goal of obtaining optical images of the reported contacts. To allow for deep water operation where surfacing for GPS position updates is impractical, and to allow for covert operations, HUGIN real-time terrain navigation is used, enabling the vehicle to update its navigation during the identification mission using a map created during the survey.

In April of 2011, FFI led a sea trial near Larvik, Norway on FFIs research vessel the H.U. Sverdrup II with participation by representatives from Canada, United States, and France. One objective of the sea trial was to acquire a data set suitable for examining incoherent and coherent change detection and automated target recognition (ATR) algorithms applied to Synthetic Aperture Sonar (SAS) imagery. The end goal is to produce an automated tool for detecting recently placed objects on the seafloor. To test these algorithms two areas were chosen, one with a comparatively benign seafloor and one with a boulder strewn complex seafloor. Each area was surveyed before and after deployment of objects. The survey time intervals varied from two days to eight days. In this paper we present the trial and show examples of SAS images and change detection of the images.

Autonomous underwater vehicles are gaining acceptance in a number of applications and countries, as a safe, cost-effective and reliable alternative to manned or remotely controlled systems. However, the actual autonomy of these vehicles is limited in many ways, restricting their potential uses. Further advances in AUV autonomy will enable new operations, such as very long endurance missions (weeks), and operations in unknown areas. While some experimentation is already taking place with e.g. under ice operations, the chance of failure is unacceptably high for many potential users. De-risking of long- endurance autonomous operations in unknown areas is thus an important goal for the AUV community. This paper gives an overview of the AUV research being carried out towards this end at the Norwegian Defence Research Establishment.