Machine Vision (MV) is essential for solving driving automation. This paper examines potential shortcomings in current MV testing strategies for highly automated driving (HAD) systems. We argue for a more comprehensive understanding of the performance factors that must be considered during the MV evaluation process, noting that neglecting these factors can lead to significant risks. This is not only relevant to MV component testing, but also to integration testing. To illustrate this point, we draw an analogy to a ship navigating towards an iceberg to show potential hidden challenges in current MV testing strategies. The main contribution is a novel framework for black-box testing which observes environmental relations. This means it is designed to enhance MV assessments by considering the attributes and surroundings of relevant individual objects. The framework provides the identification of seven general concerns about the object recognition of MV, which are not addressed adequately in established test processes. To detect these deficits based on their performance factors, we propose the use of a taxonomy called "granularity orders" along with a graphical representation. This allows an identification of MV uncertainties across a range of driving scenarios. This approach aims to advance the precision, efficiency, and completeness of testing procedures for MV.

Automated driving systems have the potential to change our transportation in the future. Since they represent safety-critical systems in an open-world context, achieving a sufficient level of safety and acceptance is a significant concern. Research and industry focus on scenario-based testing to handle the increasing verification and validation activities. A scenario catalog builds the foundation for the test cases to achieve sufficient test coverage. Although they are often not formally defined, scenario similarities and distances are commonly and implicitly used to describe and measure relationships between scenarios during testing. This paper provides an overview of existing measures used for comparing scenarios based on the dynamic behavior of their traffic participants, and how these measures are applied in testing. Therefore, a formal definition of scenario distance, similarity measures, and processable scenario types is introduced. Possible applications are identified and allocated to the scenario-based testing process. It is highlighted how the presented structured understanding of this topic helps to transfer and reuse an existing similarity or distance measure for multiple applications and to create meaningful combinations of those measures to consider all aspects of scenarios comprehensively. This facilitates a more consistent and efficient validation process.

Scenario generation is one of the essential steps in scenario-based testing and, therefore, a significant part of the verification and validation of driver assistance functions and autonomous driving systems. However, the term scenario generation is used for many different methods, e.g., extraction of scenarios from naturalistic driving data or variation of scenario parameters. This survey aims to give a systematic overview of different approaches, establish different categories of scenario acquisition and generation, and show that each group of methods has typical input and output types. It shows that although the term is often used throughout literature, the evaluated methods use different inputs and the resulting scenarios differ in abstraction level and from a systematical point of view. Additionally, recent research and literature examples are given to underline this categorization.

Mastering a specified operational design domain (ODD) and the continuous increase in automation both require comprehensive perception, prediction, and planning algorithms, thus sophisticated automated vehicle (AV) system architectures. Scenario-based testing is emerging as a promising technique attempting to efficiently run tests on all critical situations for AVs’ verification and validation. ODD, AV architectures, and scenario-based testing represent three evolving development worlds with their specific characteristics, peculiarities, and point of view, which must be consolidated and harmonized to enable a seamless design, verification, and validation process. For that reason, this paper proposes an ontology aiming at consistently reconciling the worlds of ODD, AV architectures, and scenario-based testing utilizing cross-relationships. Consequently, the ontology-based approach offers systematic guidance for an ODD definition, the design of a vehicle system architecture for automated driving, and the derivation of corresponding scenario-based test cases during the development process according to the V-model. A case study including the use case “follow lane on a motorway” for the lateral control of an active lane keeping system is exhibited, applying the introduced ontology. Therein it is also shown how the ontology-based approach enhances test efficiency and incremental development. The paper concludes by highlighting the corresponding advantages and limitations of the presented work.

Since automated driving functions are safety-critical systems, extensive validation and verification is necessary. Scenario-based testing is a promising approach for this challenge. For selection of relevant scenarios, collected data and knowledge models are potential sources. In this paper we introduce a concept to use recorded trajectory and map data, abstracted to maneuvers, to describe the scenarios and visualize them intuitively. This enables a data-driven scenario-mining process to find relevant scenarios for the testing of automated driving functions. To compare the scenarios, a similarity measure based on the manuevers is designed and the scenarios and their similarities are represented as a graph. Graph-visualization methods, already successfully applied in other domains, structure the collected data for further analysis. The concept is exemplary applied to an urban traffic dataset.

Driving scenarios are an essential part of validation of future highly automated driving (HAD) systems. In order to provide a valid proof of safety, it is crucial to test the system in as many realistic driving scenarios as possible. For this reason, it is necessary to extract driving scenarios from recorded data. A particular challenge in urban traffic is that there is a high degree of interaction between road users that needs to be considered. In this paper we present a concept for a maneuver-based extraction of driving scenarios. The extracted scenarios are provided in a format that supports a swift understanding of the content. In addition to the mere driving scenarios, parameter ranges for each scenario are grouped and aggregated from the data. Hence, the scenarios extracted with the presented concept can be used for re-simulation during the validation. We provide some results from the scenario extraction for an intersection from the INTERACTION data set.

As the level of automation and variety of Advanced Driver Assistance Systems (ADAS) and Automated Driving (AD) increases, new challenges for Verification and Validation (V&V) methods emerge. This applies especially in urban areas due to the combination of many different environmental elements, participant types, and interactions between the participants. Scenario-based testing and resimulation of recorded data are promising approaches to tackle these new challenges. An elementary component of these methods is the scenario description, which serves as a connection between different working steps in the V&V workflow. This heterogeneous usage of the scenarios during the development and validation process leads to a multitude of different, sometimes contradictory, demands on the scenario description. Nevertheless, a uniform description is desirable for easy exchange and automation. The contribution of this paper is twofold: Firstly, the described versatile field of demands is systematically broken down to requirements for the scenario description languages. This step is essential to ensure broad applicability. Secondly, this paper introduces a holistic scenario description language that is usable for generation, extraction from real-world test drives and execution of the scenarios enabling an automated workflow and increased traceability between generated, extracted and resimulated scenarios. The description and uses a model based approach and has been exemplarily tested for manually created scenarios and automatic resimulation of real-word test drives.

The validation of Advanced Driver Assistance Systems (ADAS) and Automated Driving Systems (ADS) is a major challenge for the automotive industry. Scenario based testing is a promising validation approach, but relies on a scenario database with a high coverage of possible scenarios. Especially in urban areas, the presence of different traffic objects and vulnerable road users (VRU) in combination with semi-structured environments leads to a huge number of possible scenarios. In this paper, a clustering method for grouping real-world driving sequences into semantically similar sequences is introduced. The method allows to deal with the complexity due to the varying number of traffic objects and different sequence lengths and can provide insights over realworld probabilities of driving scenarios. For comparison of different driving sequences, dynamic time warping (DTW) is leveraged. The method is applied to the vehicle trajectories of four urban intersections from the inD dataset and tested with different evaluations. Results show, that an analysis of raw trajectories provides useful insights and is an important component on the path towards a sufficient scenario database.

Ensuring the functional correctness and safety of autonomous vehicles is a major challenge for the automotive industry. However, exhaustive physical test drives are not feasible, as billions of driven kilometers would be required to obtain reliable results. Scenario-based testing is an approach to tackle this problem and reduce necessary test drives by replacing driven kilometers with simulations of relevant or interesting scenarios. These scenarios can be generated or extracted from recorded data with machine learning algorithms or created by experts. In this paper, we propose a novel graphical scenario modeling language. The graphical framework allows experts to create new scenarios or review ones designed by other experts or generated by machine learning algorithms. The scenario description is modeled as a graph and based on behavior trees. It supports different abstraction levels of scenario description during software and test development. Additionally, the graph-based structure provides modularity and reusable sub-scenarios, an important use case in scenario modeling. A graphical visualization of the scenario enhances comprehensibility for different users. The presented approach eases the scenario creation process and increases the usage of scenarios within development and testing processes.