Safeguarding autonomous vehicles is a constant challenge, since unknown circumstances that the system may not be able to handle can always arise in real-world traffic. This work proposes a monitoring framework for automotive perception sensors to detect such situations. The objective is to detect anomalous behavior from LiDAR and camera sensors at the level of object state estimations. A contrastive embedding method is used to map object states into a structured latent space. An intelligent trigger utilizes this representation space to perform anomaly detection. A key feature of the monitoring framework is that no anomaly labels are required during the training. Further, the proposed monitoring framework can be applied online, complying with ISO 21448 regarding operation phase activities. Experiments are performed on the publicly available real-world nuScenes dataset.

Simulation-based validation of autonomous driving functions relies on data-driven models capable of reproducing realistic human-like vehicle trajectories. However, most existing approaches are either not physically realistic or they lack controllability, which is essential for generating critical and rare scenarios in simulation. To address these limitations, this work proposes a physics-informed conditional generative framework designed to produce multiple, physically realistic trajectories that approximate the distribution of real driving behavior. End-to-end training is performed using the Gumbel-Softmax relaxation, allowing gradients to propagate through discrete sampling without ground-truth control supervision. Experiments on the Argoverse 1 Motion Forecasting dataset indicate that the model produces multiple physically consistent trajectories that satisfy common goal conditions, with diversity controlled via sampling temperature. A calibration procedure aligns the model’s generative spread with empirical variability observed in real trajectories, thereby helping to bridge the sim-to-real gap in driving simulations.

Roadside cameras are increasingly central to intelligent transportation systems, yet existing monocular 3D object detection methods depend on costly data collection, manual 3D annotation, and computationally heavy architectures that limit scalability. This paper introduces a lightweight framework for 3D object localization in global coordinates using only sequences of 2D detections from static infrastructure cameras. The method exploits the speed and open-set generalization of modern 2D detectors and is trained entirely on 2D–3D correspondences generated through a novel automated pipeline that projects real-world trajectories from the Waymo dataset and vehicle models from the ShapeNet dataset onto arbitrary virtual cameras. This enables zero-shot deployment on any camera with known intrinsics and extrinsics, requiring no real 3D labels. To further support infrastructure perception research, DriveInfra60, a 60-minute multi-view dataset featuring dynamic maneuvers recorded by eight synchronized high-resolution cameras with centimeter-accurate GNSS + RTK ground truth, is released. Extensive experiments demonstrate strong zero-shot performance across diverse datasets. The proposed framework offers a practical, annotation-free solution for large-scale roadside deployment, providing accurate 3D localization with minimal computational overhead.

As autonomous vehicles are rolled out, measures must be taken to ensure their safe operation. In order to supervise a system that is already in operation, monitoring frameworks are frequently employed. These run continuously online in the background, supervising the system status and recording anomalies. This work proposes an online monitoring framework to detect anomalies in object state representations. Thereby, a key challenge is creating a framework for anomaly detection without anomaly labels, which are usually unavailable for unknown anomalies. To address this issue, this work applies a self-supervised embedding method to translate object data into a latent representation space. For this, a JEPA-based self-supervised prediction task is constructed, allowing training without anomaly labels and the creation of rich object embeddings. The resulting expressive JEPA embeddings serve as input for established anomaly detection methods, in order to identify anomalies within object state representations. This framework is particularly useful for applications in real-world environments, where new or unknown anomalies may occur during operation for which there are no labels available. Experiments performed on the publicly available, real-world nuScenes dataset illustrate the framework's capabilities.

Scenario-based testing of automated driving functions requires both the clustering of traffic situations to identify representative scenarios and the detection of novel or rare events, often referred to as edge cases. Achieving these objectives demands meaningful similarity measures and expressive scenario representations that jointly capture the road infrastructure and the dynamic behavior of traffic participants. This paper presents an expert-knowledge-guided metric learning framework for traffic scenario representation. Expert knowledge is employed to define constraints on the shape of the latent space, ensuring semantic consistency and interpretability. The proposed network uses a topological representation of the road infrastructure, along with the states of the ego vehicle and its surrounding co-vehicles, to generate a joint latent embedding that captures their spatial and behavioral interactions. Positive and negative samples for metric learning are generated automatically through a graph-based data mining process, eliminating the need for manual labeling. Experiments on the openly available INTERACTION dataset show that the proposed scenario representation improves clustering quality and feature stability compared to baseline methods. This work introduces a dataset comprising scenarios recorded at a real urban intersection, along with a condensed set of scenarios on a test site to simulate driving errors and rare events in a controlled environment. The proposed model is applied to this dataset to demonstrate its effectiveness in outlier detection.

Reliable localization in GNSS-denied environments remains a fundamental challenge for intelligent vehicles, as inertial navigation systems accumulate unbounded drift without external correction. Existing approaches provide drift correction through dedicated infrastructure, expensive external sensors, or complex multi-sensor fusion, each introducing practical deployment barriers. We propose Evidential Velocity Correction using Mamba (EVC-Mamba), a learning-based architecture that transforms onboard vehicle sensor data into a virtual velocity sensor for IMU drift correction without additional hardware. A Mamba-based selective state space model captures the temporal dynamics of vehicle motion, while evidential deep learning with a Normal-Inverse-Gamma distribution provides principled uncertainty quantification. The resulting uncertainty-aware velocity estimate is incorporated as a virtual correction measurement into an Error-State Extended Kalman Filter to reduce position drift. Evaluation on real-world vehicle data demonstrates that inertial navigation using the proposed velocity correction achieves localization accuracy within 10

Approval of ADS depends on evaluating its behavior within representative real-world traffic scenarios. A common way to obtain such scenarios is to extract them from real-world data recordings. These can then be grouped and serve as basis on which the ADS is subsequently tested. This poses two central challenges: how scenarios are extracted and how they are grouped. Existing extraction methods rely on heterogeneous definitions, hindering scenario comparability. For the grouping of scenarios, rule-based or ML-based methods can be utilized. However, while modern ML-based approaches can handle the complexity of traffic scenarios, unlike rule-based approaches, they lack interpretability and may not align with domain-knowledge. This work contributes to a standardized scenario extraction based on the Scenario-as-Specification concept, as well as a domain-knowledge-guided scenario clustering process. Experiments on the highD dataset demonstrate that scenarios can be extracted reliably and that domain-knowledge can be effectively integrated into the clustering process. As a result, the proposed methodology supports a more standardized process for deriving scenario categories from highway data recordings and thus enables a more efficient validation process of automated vehicles.

Accurate and robust localization is essential for autonomous mobility systems in real-world environments. While fusing Inertial Measurement Unit (IMU) data with satellite-based correction signals provides precise vehicle pose estimates, performance degrades substantially during outages. Recent studies indicate that Machine Learning (ML) can improve IMU-based proprioceptive localization, highlighting untapped potential for onboard sensors readily available in production vehicles. This paper introduces Physics-Regularized Machine Learning for Localization (PRML2), a hybrid framework that combines the complementary strengths of Kalman filtering and data-driven learning to estimate vehicle pose directly from onboard sensors. A key aspect of PRML2 is its physics-regularized learning, enabled by end-to-end training of an ML model through a differentiable Kalman filter. This improves consistency with vehicle motion models, thereby enhancing both localization accuracy and generalization across driving conditions. We evaluate the performance limits of ML-enhanced onboard odometry on a publicly available dataset and show that PRML2 achieves superior localization accuracy and demonstrates real-time capability. This work also introduces a novel dataset to support vehicle localization research under low-friction conditions. The proposed framework provides a robust and cost-effective solution for vehicle localization under degraded sensing conditions by integrating learning with physics-based priors.

Accurate and uncertainty-aware trajectory prediction remains a core challenge for autonomous driving, driven by complex multi-agent interactions, diverse scene contexts and the inherently stochastic nature of future motion. Diffusion-based generative models have recently shown strong potential for capturing multimodal futures, yet existing approaches such as cVMD suffer from slow sampling, limited exploitation of generative diversity and brittle scenario encodings. This work introduces cVMDx, an enhanced diffusion-based trajectory prediction framework that improves efficiency, robustness and multimodal predictive capability. Through DDIM sampling, cVMDx achieves up to a 100x reduction in inference time, enabling practical multi-sample generation for uncertainty estimation. A fitted Gaussian Mixture Model further provides tractable multimodal predictions from the generated trajectories. In addition, a CVQ-VAE variant is evaluated for scenario encoding. Experiments on the publicly available highD dataset show that cVMDx achieves higher accuracy and significantly improved efficiency over cVMD, enabling fully stochastic, multimodal trajectory prediction.

This paper introduces a novel machine learning architecture for an efficient estimation of the probabilistic space-time representation of complex traffic scenarios. A detailed representation of the future traffic scenario is of significant importance for autonomous driving and for all active safety systems. In order to predict the future spacetime representation of the traffic scenario, first the type of traffic scenario is identified and then the machine learning algorithm maps the current state of the scenario to possible future states. The input to the machine learning algorithms is the current state representation of a traffic scenario, termed as the Augmented Occupancy Grid (AOG). The output is the probabilistic space-time representation which includes uncertainties regarding the behaviour of the traffic participants and is termed as the Predicted Occupancy Grid (POG). The novel architecture consists of two Stacked Denoising Autoencoders (SDAs) and a set of Random Forests. It is then compared with the other two existing architectures that comprise of SDAs and DeconvNet. The architectures are validated with the help of simulations and the comparisons are made both in terms of accuracy and computational time. Also, a brief overview on the applications of POGs in the field of active safety is presented.

Precise vehicle state estimation is crucial for safe and reliable autonomous driving. The number of measurable states and their precision offered by the onboard vehicle sensor system are often constrained by cost. For instance, measuring critical quantities such as the Vehicle Sideslip Angle (VSA) poses significant commercial challenges using current optical sensors. This paper addresses these limitations by focusing on the development of high-performance virtual sensors to enhance vehicle state estimation for active safety. The proposed Uncertainty-Aware Hybrid Learning (UAHL) architecture integrates a machine learning model with vehicle motion models to estimate VSA directly from onboard sensor data. A key aspect of the UAHL architecture is its focus on uncertainty quantification for individual model estimates and hybrid fusion. These mechanisms enable the dynamic weighting of uncertainty-aware predictions from machine learning and vehicle motion models to produce accurate and reliable hybrid VSA estimates. This work also presents a novel dataset named Real-world Vehicle State Estimation Dataset (ReV-StED), comprising synchronized measurements from advanced vehicle dynamic sensors. The experimental results demonstrate the superior performance of the proposed method for VSA estimation, highlighting UAHL as a promising architecture for advancing virtual sensors and enhancing active safety in autonomous vehicles.

Predicting the motion of traffic participants accurately remains a challenging task in the field of automated driving. Especially interactions between traffic participants introduce high complexity and interdependencies into the environment prediction. This work presents the remarkable performance of a Partially Observable Markov Decision Process (POMDP) framework to stochastically predict and safely respond to an interacting environment. The framework is validated for its ability to increase the overall Ego-Vehicle safety by preemptively triggering a de-escalation maneuver. The performance of the framework is analyzed on a publicly available dataset with real-world traffic (Argoverse) and on highly critical simulation scenarios specified by Euro-NCap for emergency braking functions. The results show quantitatively that the proposed framework significantly contributes to an early de-escalation of critical scenarios. Such an early de-escalation increases the safety and comfort of automated vehicles.

The ability to operate safely in increasingly complex traffic scenarios is a fundamental requirement for Automated Driving Systems (ADS). Ensuring the safe release of ADS functions necessitates a precise understanding of the occurring traffic scenarios. To support this objective, this work introduces a pipeline for traffic scenario clustering and the analysis of scenario category completeness. The Clustering Vector Quantized - Variational Autoencoder (CVQ-VAE) is employed for the clustering of highway traffic scenarios and utilized to create various catalogs with differing numbers of traffic scenario categories. Subsequently, the impact of the number of categories on the completeness considerations of the traffic scenario categories is analyzed. The results show an outperforming clustering performance compared to previous work. The trade-off between cluster quality and the amount of required data to maintain completeness is discussed based on the publicly available highD dataset.

Simulation is considered a critical component in the validation of autonomous driving systems, particularly for evaluating rare and safety-critical scenarios. While recent trajectory prediction methods have shown strong capabilities in generating realistic future trajectories, these models are typically designed to estimate the most likely outcome and do not support explicit conditioning on scenario parameters. As a result, their use in targeted simulation or scenario design remains limited. To address this, a framework for conditioned trajectory generation is presented, enabling the generation of a set of diverse and physically plausible trajectories from a single initial scenario state, conditioned on interpretable highlevel variables such as desired endpoints or average velocity. A hybrid machine learning framework is employed to generate control actions, which are subsequently processed by a kinematic motion model. Expert knowledge is incorporated through soft physical constraints and a smoothness loss, ensuring physical realism and signal continuity. Through experiments, it is demonstrated that realistic and diverse trajectories aligned with scenario context can be produced. The proposed framework can be integrated into simulation environments for validating autonomous vehicles.

Recent cooperative perception datasets have played a crucial role in advancing smart mobility applications by enabling information exchange between intelligent agents, helping to overcome challenges such as occlusions and improving overall scene understanding. While some existing real-world datasets incorporate both vehicle-to-vehicle and vehicle-to-infrastructure interactions, they are typically limited to a single intersection or a single vehicle. A comprehensive perception dataset featuring multiple connected vehicles and infrastructure sensors across several intersections remains unavailable, limiting the benchmarking of algorithms in diverse traffic environments. Consequently, overfitting can occur, and models may demonstrate misleadingly high performance due to similar intersection layouts and traffic participant behavior. To address this gap, we introduce UrbanIng-V2X, the first large-scale, multi-modal dataset supporting cooperative perception involving vehicles and infrastructure sensors deployed across three urban intersections in Ingolstadt, Germany. UrbanIng-V2X consists of 34 temporally aligned and spatially calibrated sensor sequences, each lasting 20 seconds. All sequences contain recordings from one of three intersections, involving two vehicles and up to three infrastructure-mounted sensor poles operating in coordinated scenarios. In total, UrbanIng-V2X provides data from 12 vehicle-mounted RGB cameras, 2 vehicle LiDARs, 17 infrastructure thermal cameras, and 12 infrastructure LiDARs. All sequences are annotated at a frequency of 10 Hz with 3D bounding boxes spanning 13 object classes, resulting in approximately 712k annotated instances across the dataset. We provide comprehensive evaluations using state-of-the-art cooperative perception methods and publicly release the codebase, dataset, HD map, and a digital twin of the complete data collection environment.

Trajectory prediction is crucial to advance autonomous driving, improving safety, and efficiency. Although end-to-end models based on deep learning have great potential, they often do not consider vehicle dynamic limitations, leading to unrealistic predictions. To address this problem, this work introduces a novel hybrid model that combines deep learning with a kinematic motion model. It is able to predict object attributes such as acceleration and yaw rate and generate trajectories based on them. A key contribution is the incorporation of expert knowledge into the learning objective of the deep learning model. This results in the constraint of the available action space, thus enabling the prediction of physically feasible object attributes and trajectories, thereby increasing safety and robustness. The proposed hybrid model facilitates enhanced interpretability, thereby reinforcing the trustworthiness of deep learning methods and promoting the development of safe planning solutions. Experiments conducted on the publicly available real-world Argoverse dataset demonstrate realistic driving behaviour, with benchmark comparisons and ablation studies showing promising results.

Realistic test scenarios are an indispensable tool in all development phases of automated driving functions. This paper presents a hybrid method that combines the advantages of data-driven methods with the modeling of realistic driving behavior using vehicle dynamics models. A neural network autoregressively generates a wide variety of starting configurations. A scenario planner within a ROS environment, which can be used to generate both virtual and real test scenarios, creates test cases from these starting configurations. The procedure is illustrated using three test scenarios, which can be found at https://doi.org/10.5281/zenodo.14002640. The neural network presented here slightly outperforms state-of-the-art models in replicating the distribution of road users in the real-world dataset. Moreover, it significantly reduces the number of scenarios where road users are positioned in the starting configuration in such a way that a collision with other road users can no longer be prevented, even if emergency braking is applied. The presented approach can play a key role in the scenario-based validation process of automated vehicles.

In order to reliably validate autonomous driving functions, known risks must be taken into account and unknown risks must be identified. This work addresses this challenge by investigating risks at the level of object state estimations. The proposed methodology utilizes the differences between object state estimations from independent sensors, enabling the detection of relevant differences. This is a significant advantage, because sensor errors can be detected without ground truth. A deep autoencoder architecture is introduced to map the differences between state estimations into a latent space. The autoencoder contains Transformer and LSTM components to effectively process signals of varying lengths. The latent space is shaped using a k-means friendly design procedure, in order to find a suitable representation for anomaly detection. Detecting anomalies is a key component in the validation process of autonomous vehicles, contributing to the identification of unknown risks. The proposed approach is evaluated using real-world automotive sensor data from cameras and laser scanners in the publicly available nuScenes dataset. The results show that the generated latent space using the k-means friendly procedure is well suited for clustering differences between state estimations from these two sensors and thus for anomaly detection. In the framework specified in the safety standard ISO 21448 (SOTIF) the proposed methodology can play a key role for the detection of unknown risks on the perception level during the operation phase of autonomous vehicles.

This work introduces the conditioned Vehicle Motion Diffusion (cVMD) model, a novel network architecture for highway trajectory prediction using diffusion models. The proposed model ensures the drivability of the predicted trajectory by integrating non-holonomic motion constraints and physical constraints into the generative prediction module. Central to the architecture of cVMD is its capacity to perform uncertainty quantification, a feature that is crucial in safety-critical applications. By integrating the quantified uncertainty into the prediction process, the cVMD's trajectory prediction performance is improved considerably. The model's performance was evaluated using the publicly available highD dataset. Experiments show that the proposed architecture achieves competitive trajectory prediction accuracy compared to state-of-the-art models, while providing guaranteed drivable trajectories and uncertainty quantification.

This research focuses on solving challenges related to identifying unfamiliar object categories in the realm of Open-Set Object Detection (OSOD) using infrastructure sensors. Traditional camera-based OSOD systems struggle to generate proposals for dissimilar novel classes due to a lack of feature similarity. This research introduces a novel approach named Fusion Object Detector (FOD), which emphasizes the localization and identification of semantically dissimilar unknown objects through a multimodal fusion architecture involving infrastructure-mounted cameras and LiDARs. FOD leverages a camera-based closed-set object detector for the identification of known class objects, while simultaneously utilizing clusters derived from fused LiDAR point clouds for the detection of unknown class objects. This research work also presents a novel dataset named Thermal camera and LiDAR in Infrastructure Dataset (TLID). TLID comprises fused sensor measurements from multiple thermal cameras and LiDARs mounted in three urban crossings of Ingolstadt city and at CARISSMA outdoor test track. The proposed methodology is evaluated using both an in-house dataset and a publicly available infrastructure dataset for the task of OSOD. The results quantify the importance of multimodal sensor information for the task of identifying dissimilar unknown objects.