Safety and assurance cases risk becoming detached from the understanding needed for responsible engineering and governance decisions. More broadly, the production and evaluation of critical socio-technical systems increasingly face an understanding challenge: pressures for increased tempo, reduced scrutiny, software complexity, and growing use of AI generated artefacts may produce outputs that appear coherent without supporting genuine human comprehension. We argue that understanding should become an explicit, assessable, and defensible component of decision making: what developers, assessors, and decision makers grasp about system behavior, evidence, assumptions, risks, and residual uncertainty. Drawing on Catherine Elgin's epistemology of understanding, we outline a conceptual foundation and then use Assurance 2.0 as an engineering route to operationalize using structured argumentation, evidence, confidence, defeaters, and theory based automation. This leads to two linked artefacts: an Understanding Basis, which justifies why available understanding is sufficient for a decision, and a Personal Understanding Statement, through which participants make their grasp explicit and challengeable. We also identify risks that automation may improve artefact production while weakening understanding, and we propose initial directions for evaluating both efficacy and epistemic impact.

Confidence is central to safety and assurance cases: how much confidence a decision requires and how much the argument actually provides are both important questions. We present a new method for assessing probabilistic confidence in assurance case arguments that is simple, systematic and sound. It exploits the ways claims are decomposed in a structured argument and provides different approaches according to the different degrees of (in)dependence and diversity among subclaims and the way they eliminate concerns that undermine confidence in their parent claims. The method uses only elementary probabilistic constructions that are well-known in other contexts (e.g., Frechet bounds) but we interpret and apply them in a manner that is specifically focused on assurance arguments and requires no background in probabilistic analysis. We show that the method is not susceptible to the counterexamples that Graydon and Holloway exhibit for other approaches to confidence and we recommend it as an additional tool in evaluation of Assurance 2.0 arguments. The primary evaluation criteria for Assurance 2.0 remain logical indefeasibility and dialectical examination, but probabilistic assessment can be useful in evaluating cost/confidence tradeoffs for different risk levels, and the overall balance of confidence across a structured argument.

Decision makers need sufficient understanding to make good decisions about complex AI systems. However, AI deployment decisions are increasingly made under time-pressure, and this combined with the use of AI generated artefact creation, can mean that the existence of safety cases and system cards may no longer demonstrate that sufficient understanding exists. Our provisional methodology for making understanding explicit and assessable requires the production of an explicit description of 4 objects of understanding (decision, decision-frame, safety justification, system-in-context) and a justification for the adequacy of this understanding. In addition, the methodology provides a mechanism for describing and evaluating the adequacy of the decision-maker representation of this understanding. It builds on recent developments in safety cases using the Assurance 2.0 framework to operationalise the philosophical basis of understanding from Elgin and Arendt. To assess the methodology we trialled two different scenarios. One scenario, which we investigated through role-based analysis, concerned the risk of scheming in the deployment of an AI coding agent in a robotics company and the other scenario was for the higher uncertainty, more decision-critical argument of 'If Anyone Builds It, Everyone Dies' (Yudkowsky and Soares). The trial's central finding, for these two scenarios, is that the methodology could be applied and was found to be generative: we found the analyses that justify sufficiency of understanding (internal coherence, tethering, felicitous falsehoods, external coherence) drives the engineering.

We present a new method for assessing probabilistic confidence in assurance case arguments that is simple, systematic and sound. It exploits the ways claims are decomposed in a structured argument and provides different approaches according to the different degrees of (in)dependence and diversity among subclaims, and how they eliminate the concerns that undermine confidence in their parent claims. The method uses only elementary probabilistic constructions that are well-known in other contexts (e.g., Fréchet bounds) but we interpret and apply them in a manner that is specifically focused on assurance arguments and requires no background in probabilistic analysis. We show that the method is not susceptible to counterexamples that Graydon and Holloway exhibit for other approaches to quantifying confidence [12] and we recommend it as an additional tool and point of view in assessing Assurance 2.0 arguments that can contribute to dialectical examination and evaluation of cost/confidence tradeoffs for different risk levels and the overall balance of confidence across a structured argument.

As systems become increasingly composed of AI/ML elements, the focus of assurance tends to shift from “safe use of AI” to “safe AI.” In this position paper, we argue this is opposite to what is required and urge a systems engineering approach.

We draw on our experience working on system and software assurance and evaluation for systems important to society to summarise how safety engineering is performed in traditional critical systems, such as aircraft flight control. We analyse how this critical systems perspective might support the development and implementation of AI Safety Frameworks. We present the analysis in terms of: system engineering, safety and risk analysis, and decision analysis and support. We consider four key questions: What is the system? How good does it have to be? What is the impact of criticality on system development? and How much should we trust it? We identify topics worthy of further discussion. In particular, we are concerned that system boundaries are not broad enough, that the tolerability and nature of the risks are not sufficiently elaborated, and that the assurance methods lack theories that would allow behaviours to be adequately assured. We advocate the use of assurance cases based on Assurance 2.0 to support decision making in which the criticality of the decision as well as the criticality of the system are evaluated. We point out the orders of magnitude difference in confidence needed in critical rather than everyday systems and how everyday techniques do not scale in rigour. Finally we map our findings in detail to two of the questions posed by the FAISC organisers and we note that the engineering of critical systems has evolved through open and diverse discussion. We hope that topics identified here will support the post-FAISC dialogues.

Ensuring the reliability and safety of automated decision-making is crucial. It is well-known that data distribution shifts in machine learning can produce unreliable outcomes. This paper proposes a new approach for measuring the reliability of predictions under distribution shifts. We analyze how the outputs of a trained neural network change using clustering to measure distances between outputs and class centroids. We propose this distance as a metric to evaluate the confidence of predictions under distribution shifts. We assign each prediction to a cluster with centroid representing the mean softmax output for all correct predictions of a given class. We then define a safety threshold for a class as the smallest distance from an incorrect prediction to the given class centroid. We evaluate the approach on the MNIST and CIFAR-10 datasets using a Convolutional Neural Network and a Vision Transformer, respectively. The results show that our approach is consistent across these data sets and network models, and indicate that the proposed metric can offer an efficient way of determining when automated predictions are acceptable and when they should be deferred to human operators given a distribution shift.

A traditional assurance case employs a positive argument in which reasoning steps, grounded on evidence and assumptions, sustain a top claim that has external significance. Human judgement is required to check the evidence, the assumptions, and the narrative justifications for the reasoning steps; if all are assessed good, then the top claim can be accepted. A valid concern about this process is that human judgement is fallible and prone to confirmation bias. The best defense against this concern is vigorous and skeptical debate and discussion in the manner of a dialectic or Socratic dialog. There is merit in recording aspects of this discussion for the benefit of subsequent developers and assessors. Defeaters are a means doing this: they express doubts about aspects of the argument and can be developed into subcases that confirm or refute the doubts, and can record them as documentation to assist future consideration. This report describes how defeaters, and multiple levels of defeaters, should be represented and assessed in Assurance 2.0 and its Clarissa/ASCE tool support. These mechanisms also support eliminative argumentation, which is a contrary approach to assurance, favored by some, that uses a negative argument to refute all reasons why the top claim could be false.

Assurance cases offer a structured way to present arguments and evidence for certification of systems where safety and security are critical. However, creating and evaluating these assurance cases can be complex and challenging, even for systems of moderate complexity. Therefore, there is a growing need to develop new automation methods for these tasks. While most existing assurance case tools focus on automating structural aspects, they lack the ability to fully assess the semantic coherence and correctness of the assurance arguments.In prior work, we introduced the Assurance 2.0 framework that prioritizes the reasoning process, evidence utilization, and explicit delineation of counter-claims (defeaters) and counter-evidence. In this paper, we present our approach to enhancing Assurance 2.0 with semantic rule-based analysis capabilities using common-sense reasoning and answer set programming solvers, specifically s(CASP). By employing these analysis techniques, we examine the unique semantic aspects of assurance cases, such as logical consistency, adequacy, indefeasibility, etc. The application of these analyses provides both system developers and evaluators with increased confidence about the assurance case.

All interactive systems need a model of their world that they can use to calculate effective behavior. For assurance, the model needs to be accurate but, in autonomous vehicles and many other AI applications, the model is built by a perception system based on machine learning and the dependability perspective maintains that its accuracy cannot be assured. We outline this perspective and methods for providing assurance using guards and defense in depth, and we also outline predictive processing as a possible way to construct assured models. We then discuss LLMs, which typically lack explicit models of the world, and suggest possible mitigations for their correspondingly unpredictable behavior. Finally, we consider models in AGI.

We outline the principles of classical assurance for computer-based systems that pose significant risks. We then consider application of these principles to systems that employ Artificial Intelligence (AI) and Machine Learning (ML). A key element in this "dependability" perspective is a requirement to have near-complete understanding of the behavior of critical components, and this is considered infeasible for AI and ML. Hence the dependability perspective aims to minimize trust in AI and ML elements by using "defense in depth" with a hierarchy of less complex systems, some of which may be highly assured conventionally engineered components, to "guard" them. This may be contrasted with the "trustworthy" perspective that seeks to apply assurance to the AI and ML elements themselves. In cyber-physical and many other systems, it is difficult to provide guards that do not depend on AI and ML to perceive their environment (e.g., other vehicles sharing the road with a self-driving car), so both perspectives are needed and there is a continuum or spectrum between them. We focus on architectures toward the dependability end of the continuum and invite others to consider additional points along the spectrum. For guards that require perception using AI and ML, we examine ways to minimize the trust placed in these elements; they include diversity, defense in depth, explanations, and micro-ODDs. We also examine methods to enforce acceptable behavior, given a model of the world. These include classical cyber-physical calculations and envelopes, and normative rules based on overarching principles, constitutions, ethics, or reputation. We apply our perspective to autonomous systems, AI systems for specific functions, generic AI such as Large Language Models, and to Artificial General Intelligence (AGI), and we propose current best practice and an agenda for research.

An assurance case should provide justifiable confidence in the truth of a claim about some critical property of a system or procedure, such as safety or security. We consider how confidence can be assessed in the rigorous approach we call Assurance 2.0. Our goal is indefeasible confidence and we approach it from four different perspectives: logical soundness, probabilistic assessment, dialectical examination, and residual risks.

This study introduces the Misclassification Likelihood Matrix (MLM) as a novel tool for quantifying the reliability of neural network predictions under distribution shifts. The MLM is obtained by leveraging softmax outputs and clustering techniques to measure the distances between the predictions of a trained neural network and class centroids. By analyzing these distances, the MLM provides a comprehensive view of the model's misclassification tendencies, enabling decision-makers to identify the most common and critical sources of errors. The MLM allows for the prioritization of model improvements and the establishment of decision thresholds based on acceptable risk levels. The approach is evaluated on the MNIST dataset using a Convolutional Neural Network (CNN) and a perturbed version of the dataset to simulate distribution shifts. The results demonstrate the effectiveness of the MLM in assessing the reliability of predictions and highlight its potential in enhancing the interpretability and risk mitigation capabilities of neural networks. The implications of this work extend beyond image classification, with ongoing applications in autonomous systems, such as self-driving cars, to improve the safety and reliability of decision-making in complex, real-world environments.

Assurance cases are gaining traction as a viable means of certification in various safety/security critical industries. In this paper, we introduce a theory-based, property-driven continuous assurance framework that eliminates ad-hoc case constructions through emphasis on validity and soundness of the arguments, confidence of the claims/arguments/evidences and the systematized specification of defeaters. We then describe tools and automation support for semantic analysis of assurance cases and their synthesis. Finally, we showcase a continuous assurance tools infrastructure through an example.

The use of assurance cases is gaining popularity, particularly in the safety-critical system industry, as an organized approach to submitting documentation for the safety and security certification of systems. However, these arguments can become overwhelming and complicated, even for moderately complex systems. Therefore, there is a compelling requirement to develop new automation strategies that can aid in creating and assessing assurance cases. Existing assurance-case tools primarily automate syntactic analysis, focusing on structural completeness, while providing limited or no support for semantically evaluating the logical aspects of the assurance case. In prior work, we introduced a framework called Assurance 2.0, which aims to enhance the rigor of assurance cases by emphasizing the reasoning process, evidence utilized, and explicit identification of counter-claims (defeaters) and counter-evidence. In this paper, we present a new approach to enhancing Assurance 2.0 by incorporating semantic rule-based analysis capabilities. Firstly, we systematically convert the assurance case into Prolog predicates and constraints. Then, leveraging the analysis capabilities of the s(CASP), a goal-directed top-down solver for Constraints Answer Set Programs, we evaluate the semantic properties of assurance cases, including logical consistency, completeness, and indefeasibility. The application of these analyses provides both authors and evaluators with higher confidence when assessing the assurance case.

Assurance cases are gaining traction as a means of certification in Aerospace and other safety and security critical industries. In this paper, we first introduce a rigorous Assurance 2.0 framework that eliminates adhoc construction of assurance cases with emphasis on the validity and soundness of the argumentation process, confidence of the claims/arguments/evidences and the systematic exploration of defeaters. We next describe the tools and automation support for Assurance 2.0 that was developed in the Clarissa project and finally highlight the key capabilities in the context of an illustrative example.

An assurance case is intended to provide justifiable confidence in the truth of its top claim, which typically concerns safety or security. A natural question is then "how much" confidence does the case provide? We argue that confidence cannot be reduced to a single attribute or measurement. Instead, we suggest it should be based on attributes that draw on three different perspectives: positive, negative, and residual doubts. Positive Perspectives consider the extent to which the evidence and overall argument of the case combine to make a positive statement justifying belief in its claims. We set a high bar for justification, requiring it to be indefeasible. The primary positive measure for this is soundness, which interprets the argument as a logical proof. Confidence in evidence can be expressed probabilistically and we use confirmation measures to ensure that the "weight" of evidence crosses some threshold. In addition, probabilities can be aggregated from evidence through the steps of the argument using probability logics to yield what we call probabilistic valuations for the claims. Negative Perspectives record doubts and challenges to the case, typically expressed as defeaters, and their exploration and resolution. Assurance developers must guard against confirmation bias and should vigorously explore potential defeaters as they develop the case, and should record them and their resolution to avoid rework and to aid reviewers. Residual Doubts: the world is uncertain so not all potential defeaters can be resolved. We explore risks and may deem them acceptable or unavoidable. It is crucial however that these judgments are conscious ones and that they are recorded in the assurance case. This report examines the perspectives in detail and indicates how Clarissa, our prototype toolset for Assurance 2.0, assists in their evaluation.

This report documents safety assurance argument templates to support the deployment and operation of autonomous systems that include machine learning (ML) components. The document presents example safety argument templates covering: the development of safety requirements, hazard analysis, a safety monitor architecture for an autonomous system including at least one ML element, a component with ML and the adaptation and change of the system over time. The report also presents generic templates for argument defeaters and evidence confidence that can be used to strengthen, review, and adapt the templates as necessary. This report is made available to get feedback on the approach and on the templates. This work was sponsored by the UK Dstl under the R-cloud framework.