Autonomous Surface Vessels (ASVs) rely on advanced perception algorithms to accurately represent internal and external conditions. One of the most challenging tasks is tracking surrounding objects, especially in the presence of model and measurement uncertainties. In case of multiple dynamic obstacles, the tracking problem becomes highly important to guarantee safe navigation. Therefore, we propose a multi-target tracking algorithm based on Light Detection And Ranging (LiDAR) and Automatic Identification System (AIS) data. To estimate the shape of other vessels, a Numerically Stable Direct Least-Squares (NSDLS) ellipse fitting algorithm is applied to LiDAR point clouds. For that purpose, the point clouds are separated into clusters using a modified version of Density-Based Spatial Clustering of Applications with Noise (DBSCAN). To improve the robustness of the tracking algorithm, the LiDAR-generated estimations are fused with AIS measurements using a Kalman Filter (KF). The motion model used for the KF enables the prediction of dynamic obstacles surrounding the ASV. This proactive awareness can be used for predictive control concepts such as Model Predictive Control. However, testing such advanced algorithms on real systems can be dangerous. A Digital Twin (DT) allows for an extendable integration of physics-based models, sensor data, and intelligent algorithms, enabling simulations in a safe environment. This work demonstrates the proposed multi-target tracking approach in a DT of an ASV, which contains vessel dynamics, sensor models, and real-world AIS data streams.

Many autonomous systems face safety challenges, requiring robust closed-loop control to handle physical limitations and safety constraints. Real-world systems, like autonomous ships, encounter nonlinear dynamics and environmental disturbances. Reinforcement learning is increasingly used to adapt to complex scenarios, but standard frameworks ensuring safety and stability are lacking. Predictive Safety Filters (PSF) offer a promising solution, ensuring constraint satisfaction in learning-based control without explicit constraint handling. This modular approach allows using arbitrary control policies, with the safety filter optimizing proposed actions to meet physical and safety constraints. We apply this approach to marine navigation, combining RL with PSF on a simulated Cybership II model. The RL agent is trained on path following and collision avpodance, while the PSF monitors and modifies control actions for safety. Results demonstrate the PSF's effectiveness in maintaining safety without hindering the RL agent's learning rate and performance, evaluated against a standard RL agent without PSF.

This article presents an algorithmic framework for real-time condition monitoring and state forecasting using multivariate data demonstrated on thermal imagery data of a ship’s engine. The proposed method aims to improve the accuracy, efficiency, and robustness of condition monitoring and state predictions by identifying the most informative sampling locations of high-dimensional datasets and extracting the underlying dynamics of the system. The method is based on a combination of Proper Orthogonal Decomposition (POD), Optimal Sampling Location (OSL), and Dynamic Mode Decomposition (DMD), allowing the identification of key features in the system’s behavior and predicting future states. Based on thermal imagery data, it is shown how thermal areas of interest can be classified via POD. By extracting the POD modes of the data, dimensions can be drastically reduced and via OSL, optimal sampling locations are found. In addition, nonlinear kernel-based Support Vector Regression (SVR) is used to build models between the optimal locations, enabling the imputation of erroneous data to improve the overall robustness. To build predictive data-driven models, DMD is applied on the subspace obtained by OSL, which leads to an intensive lower demand of computational resources, making the proposed method real-time applicable. Furthermore, an unsupervised approach for anomaly detection is proposed using OSL. The anomaly detection framework is coupled with the state prediction framework, which extends the capabilities to real-time anomaly predictions. In summary, this study proposes a robust predictive condition monitoring framework for real-time risk assessment.

This paper explores the development and practical application of a predictive digital twin specifically designed for condition monitoring, using advanced mathematical models and thermal imaging techniques. Our work presents a comprehensive approach to integrating Proper Orthogonal Decomposition (POD), Robust Principal Component Analysis (RPCA), and Dynamic Mode Decomposition (DMD) to establish a robust predictive digital twin framework. We employ these methods in a real-time experimental setup involving a heated plate monitored through thermal imaging. This system effectively demonstrates the digital twin's capabilities in real-time predictions, condition monitoring, and anomaly detection. Additionally, we introduce the use of a human-machine interface that includes virtual reality, enhancing user interaction and system understanding. The primary contributions of our research lie in the demonstration of these advanced techniques in a tangible setup, showcasing the potential of digital twins to transform industry practices by enabling more proactive and strategic asset management.

Autonomous surface vessels (ASVs) are becoming increasingly significant in enhancing the safety and sustainability of maritime operations. To ensure the reliability of modern control algorithms utilized in these vessels, digital twins (DTs) provide a robust framework for conducting safe and effective simulations within a virtual environment. Digital twins are generally classified on a scale from 0 to 5, with each level representing a progression in complexity and functionality: Level 0 (Standalone) employs offline modeling techniques; Level 1 (Descriptive) integrates sensors and online modeling to enhance situational awareness; Level 2 (Diagnostic) focuses on condition monitoring and cybersecurity; Level 3 (Predictive) incorporates predictive analytics; Level 4 (Prescriptive) embeds decision-support systems; and Level 5 (Autonomous) enables advanced functionalities such as collision avoidance and path following. These digital representations not only provide insights into the vessel's current state and operational efficiency but also predict future scenarios and assess life endurance. By continuously updating with real-time sensor data, the digital twin effectively corrects modeling errors and enhances decision-making processes. Since DTs are key enablers for complex autonomous systems, this paper introduces a comprehensive methodology for establishing a digital twin framework specifically tailored for ASVs. Through a detailed literature survey, we explore existing state-of-the-art enablers across the defined levels, offering valuable recommendations for future research and development in this rapidly evolving field.

This article proposes an approach for collision avoidance, path following, and anti-grounding of autonomous surface vessels under consideration of environmental forces based on Nonlinear Model Predictive Control (NMPC). Artificial Potential Fields (APFs) set the foundation for the cost function of the optimal control problem in terms of collision avoidance and anti-grounding. Depending on the risk of a collision given by the resulting force of the APFs, the controller optimizes regarding an adapted heading and travel speed by additionally following a desired path. For this purpose, nonlinear vessel dynamics are used for the NMPC. To extend the situational awareness concerning environmental disturbances impacted by wind, waves, and sea currents, a nonlinear disturbance observer is coupled to the entire NMPC scheme, allowing for the correction of an incorrect vessel motion due to external forces. In addition, the most essential rules according to the Convention on the International Regulations for Preventing Collisions at Sea (COLREGs) are considered. The results of the simulations show that the proposed framework can control an autonomous surface vessel under various challenging scenarios, including environmental disturbances, to avoid collisions and follow desired paths.

Autonomous surface vessels (ASVs) play an increasingly important role in the safety and sustainability of open sea operations. Since most maritime accidents are related to human failure, intelligent algorithms for autonomous collision avoidance and path following can drastically reduce the risk in the maritime sector. A DT is a virtual representative of a real physical system and can enhance the situational awareness (SITAW) of such an ASV to generate optimal decisions. This work builds on an existing DT framework for ASVs and demonstrates foundations for enabling predictive, prescriptive, and autonomous capabilities. In this context, sophisticated target tracking approaches are crucial for estimating and predicting the position and motion of other dynamic objects. The applied tracking method is enabled by real-time automatic identification system (AIS) data and synthetic light detection and ranging (LiDAR) measurements. To guarantee safety during autonomous operations, we applied a predictive safety filter to correct inputs from a reinforcement learning-based controller. The approaches are implemented into a DT built with the Unity game engine. As a result, this work demonstrates the potential of a DT capable of making predictions, playing through various what-if scenarios, and providing optimal control decisions according to its enhanced SITAW.

In the current data-intensive era, big data has become a significant asset for Artificial Intelligence (AI), serving as a foundation for developing data-driven models and providing insight into various unknown fields. This study navigates through the challenges of data uncertainties, storage limitations, and predictive data-driven modeling using big data. We utilize Robust Principal Component Analysis (RPCA) for effective noise reduction and outlier elimination, and Optimal Sensor Placement (OSP) for efficient data compression and storage. The proposed OSP technique enables data compression without substantial information loss while simultaneously reducing storage needs. While RPCA offers an enhanced alternative to traditional Principal Component Analysis (PCA) for high-dimensional data management, the scope of this work extends its utilization, focusing on robust, data-driven modeling applicable to huge data sets in real-time. For that purpose, Long Short-Term Memory (LSTM) networks, a type of recurrent neural network, are applied to model and predict data based on a low-dimensional subset obtained from OSP, leading to a crucial acceleration of the training phase. LSTMs are feasible for capturing long-term dependencies in time series data, making them particularly suited for predicting the future states of physical systems on historical data. All the presented algorithms are not only theorized but also simulated and validated using real thermal imaging data mapping a ship's engine.

This paper proposes a robust disturbance observer framework for maritime autonomous surface vessels considering model and measurement uncertainties. The core contribution lies in a nonlinear disturbance observer, reconstructing the forces on a vessel impacted by the environment. For this purpose, mappings are found leading to synchronized global exponentially stable error dynamics. With the stability theory of Lyapunov, it is proven that the error converges exponentially into a ball, even if the disturbances are highly dynamic. Since measurements are affected by noise and physical models can be erroneous, an unscented Kalman filter (UKF) is used to generate more reliable state estimations. In addition, a noise estimator is introduced, which approximates the noise strength. Depending on the severity of the measurement noise, the observed disturbances are filtered through a cascaded structure consisting of a weighted moving average (WMA) filter, a UKF, and the proposed disturbance observer. To investigate the capability of this observer framework, the environmental disturbances are simulated dynamically under consideration of different model and measurement uncertainties. It can be seen that the observer framework can approximate dynamical forces on a vessel impacted by the environment despite using a low measurement sampling rate, an erroneous model, and noisy measurements.

Multiple studies show that humans are the leading cause of maritime accidents. Therefore, the interest in introducing autonomous surface vessels, particularly in the commercial sector, is receiving increasing attention. Digital twins, virtual representations of physics assets or processes enabled through data and models for real-time control, optimization, monitoring, and informed decision-making, are seen as key enablers for autonomy. Based on their capability, a digital twin is generally categorized on a scale from 0–5 (0-standalone, 1-descriptive, 2-diagnostic, 3-predictive, 4-prescriptive, 5-autonomous). The current work aims at concretizing the categorization in the context of autonomous vessels. It demonstrates some capability levels by introducing the development of an extendable digital twin framework created in the Unity game engine using physics-based models and real-time data. For this purpose, real-time weather and automatic identification system (AIS) data is streamed. While computer-aided design (CAD) and a three-dimensional (3D) height map build the foundation of a standalone digital twin, real-time data, physics-based models, and synthetic sensors set the basis of descriptive representations. Finally, an environmental disturbance observer enables the first diagnostic abilities, while a reinforcement learning algorithm pushes towards autonomous capabilities. It is argued that the work will improve the overall situational awareness for safe sea operations.