[Context and Motivation] The automotive industry is moving towards increased automation, where features such as automated driving systems typically include machine learning (ML), e.g. in the perception system. [Question/Problem] Ensuring safety for systems partly relying on ML is challenging. Different approaches and frameworks have been proposed, typically where the developer must define quantitative and/or qualitative acceptance criteria, and ensure the criteria are fulfilled using different methods to improve e.g., design, robustness and error detection. However, there is still a knowledge gap between quality methods and metrics employed in the ML domain and how such methods can contribute to satisfying the vehicle level safety requirements. [Principal Ideas/Results] In this paper, we argue the need for connecting available ML quality methods and metrics to the safety lifecycle and explicitly show their contribution to safety. In particular, we analyse Out-of-Distribution (OoD) detection, e.g., the frequency of novelty detection, and show its potential for multiple safety-related purposes. I.e., as (a) an acceptance criterion contributing to the decision if the software fulfills the safety requirements and hence is ready-for-release, (b) in operational design domain selection and expansion by including novelty samples into the training/development loop, and (c) as a run-time measure, e.g., if there is a sequence of novel samples, the vehicle should consider reaching a minimal risk condition. [Contribution] This paper describes the possibility to use OoD detection as a safety measure, and the potential contributions in different stages of the safety lifecycle.

During the last decade, there has been significant increase in research focused on automated vehicles (AVs) and ensuring safe operation of these vehicles. However, challenges still remain, some involving the cooperation and collaboration of multiple AVs, including when and how to perform a minimal risk manoeuvre (MRM), leading to a minimal risk condition (MRC) when an AV within one of these systems is unable to complete its original goal. As most literature is focused on individual AVs, there is a need to adapt and extend the knowledge and techniques to these new contexts. Based on existing knowledge of individual AVs, this paper explores MRM strategies involving cooperative and collaborative AV systems with different capabilities. Specifically, collaborative systems have the potential to enact local MRCs, allowing continued productivity despite having one (or several) of its constituents encounter a fault. Definitions are provided for local and global MRCs, alongside discussions of their implications for MRMs. Illustrative examples are also presented for each type of system.

Safety measures need to be systemically investigated to what extent they evaluate the intended performance of Deep Neural Networks (DNNs) for critical applications. Due to a lack of verification methods for high-dimensional DNNs, a trade-off is needed between accepted performance and handling of out-of-distribution (OOD) samples.This work evaluates rejecting outputs from semantic segmentation DNNs by applying a Mahalanobis distance (MD) based on the most probable class-conditional Gaussian distribution for the predicted class as an OOD score. The evaluation follows three DNNs trained on the Cityscapes dataset and tested on four automotive datasets and finds that classification risk can drastically be reduced at the cost of pixel coverage, even when applied on unseen datasets. The applicability of our findings will support legitimizing safety measures and motivate their usage when arguing for safe usage of DNNs in automotive perception.

We have yet to see wide deployment of automated driving systems (ADSs) on public roads. One of the reasons is the challenge of ensuring the systems’ safety. The operational design domain (ODD) can be used to confine the scope of the ADS and subsequently also its safety case. For this to be valid the ADS needs to have strategies to remain in the ODD throughout its operations. In this paper we discuss the role of the minimal risk condition (MRC) as a means to ensure this. Further, we elaborate on the need for hierarchies of MRCs to cope with diverse system degradations during operations.

Machine learning (ML)-enabled approaches are considered a substantial support technique of detection and classification of obstacles of traffic participants in self-driving vehicles. Major breakthroughs have been demonstrated the past few years, even covering complete end-to-end data processing chain from sensory inputs through perception and planning to vehicle control of acceleration, breaking and steering. YOLO (you-only-look-once) is a state-of-the-art perception neural network (NN) architecture providing object detection and classification through bounding box estimations on camera images. As the NN is trained on well annotated images, in this paper we study the variations of confidence levels from the NN when tested on hand-crafted occlusion added to a test set. We compare regular pedestrian detection to upper and lower body detection. Our findings show that the two NN using only partial information perform similarly well like the NN for the full body when the full body NN's performance is 0.75 or better. Furthermore and as expected, the network, which is only trained on the lower half body is least prone to disturbances from occlusions of the upper half and vice versa.

One of the biggest challenges for self-driving road vehicles is how to argue that their safety cases are complete. The operational design domain (ODD) of the automated driving system (ADS) can be u ...

One of the major challenges of designing automated driving systems (ADS) is showing that they are safe. This includes safety analysis of interactions between humans and the ADS, a multidisciplinary task involving functional safety and human factors expertise. In this paper, we lay the foundation for a safety analysis method for these interactions, which builds upon combining human factors knowledge with known techniques from the functional safety domain. The aim of the proposed method is finding safety issues in proposed HMI protocols. It combines constructing interaction sequences between human and ADS as a variant of sequence diagrams , and use these sequences as input to a cause-consequence analysis with the purpose of finding potential interaction faults that may lead to dangerous failures. Based on a this analysis, the HMI design can be improved to reduce safety risks, and the analysis results can also be used as part of the ADS safety case.

With the advent of automated cooperative systems, new challenges are raised with respect to fulfilling safety. This paper presents a novel approach in analysing and assessment of the risks for such systems. Moreover, a distributed conceptual architecture is proposed and discussed. This architecture features separated layers of concern for strategic, tactical and operational functions. We presented how the tactical layer can be allocated to either centralised or distributed versions of the architecture. An item definition, i.e. a description of the concept according to the ISO 26262 framework, for the platooning system, is presented together with a description of the operational design domain. A preliminary architecture for a functional safety concept of the item is presented. Finally, we present a risk analysis of the item according to a novel approach using a quantitative risk norm and incident classification, instead of traditional hazard analysis and risk assessment.

Deep Neural Networks (DNN) have improved the quality of several non-safety related products in the past years. However, before DNNs should be deployed to safety-critical applications, their robustness needs to be systematically analyzed. A common challenge for DNNs occurs when input is dissimilar to the training set, which might lead to high confidence predictions despite proper knowledge of the input. Several previous studies have proposed to complement DNNs with a supervisor that detects when inputs are outside the scope of the network. Most of these supervisors, however, are developed and tested for a selected scenario using a specific performance metric. In this work, we emphasize the need to assess and compare the performance of supervisors in a structured way. We present a framework constituted by four datasets organized in six test cases combined with seven evaluation metrics. The test cases provide varying complexity and include data from publicly available sources as well as a novel dataset consisting of images from simulated driving scenarios. The latter we plan to make publicly available. Our framework can be used to support DNN supervisor evaluation, which in turn could be used to motive development, validation, and deployment of DNNs in safety-critical applications.

The presented paper presents the ongoing Swedish national research project FUSE (FUnctional Safety and Evolvable architectures for autonomy). Some of the research questions addressed in this project are summarized. The research questions are related both to functional safety and the E/E architecture of vehicles aimed for higher degrees of automation, including fully autonomous ones.

When arguing safety for an autonomous road vehicle it is considered very hard to show that the sensing capability is sufficient for all possible scenarios that might occur. Already for today's manually driven road vehicles equipped with advanced driver assistance systems (ADAS), it is far from trivial how to argue that the sensor systems are sufficiently capable of enabling a safe behavior. In this paper, we argue that the transition from ADAS to automated driving systems (ADS) enables new solution patterns for the safety argumentation dependent on the sensor systems. A key factor is that the ADS itself can compensate for a lower sensor capability, by for example lowering the speed or increasing the distances. The proposed design strategy allocates safety requirements on the sensors to determine their own capability. This capability is then to be balanced by the tactical decisions of the ADS equipped road vehicle.

The introduction of highly automated driving and autonomous road vehicles will imply new functional safety challenges. The higher complexity and the partly implicit definition of the tasks for the E/E systems will make it harder to argue completeness and correctness of the safety requirements in each phase of the ISO 26262 lifecycle. This paper discusses the new situation in terms of an increasing semantic gap, and recommends to perform more safety refinement steps. As a consequence, ISO 26262 should be amended with activities prescribing new refinements levels.

The software testing state of practice is not as good as it ought to be. To reduce the gap between the state of practice and the state of the art, improvement in software testing is necessary. Existing improvement models focus too little on, or lack important aspects of, software testing. Therefore a new model is required. This paper describes a test improvement model called TIM. The model functions as a guidebook for test improvement. It is mainly inspired by SEI's Capability Maturity Model and Gelperin's Testability Maturity Model. TIM introduces new perspectives to test improvement, with explicit focus on cost‐effectiveness and risk management. The model addresses five ‘key areas’ of testing: organization, planning and tracking, test cases, testware and reviews. TIM also includes an assessment procedure. The model is continuously revised as knowledge is gained through research and experience of use. This paper presents results from three applications of TIM. © 1997 John Wiley & Sons, Ltd.