Abstract This chapter outlines the System Design Plan, which aims to develop a system that meets functional safety and regulatory requirements while ensuring reliability, transparency, and stakeholder satisfaction. It emphasizes the importance of risk management, effective integration of subsystems, and continuous stakeholder engagement to deliver a compliant and trustworthy system design. The chapter also explores agile adaptations and the role of stakeholder feedback in refining the system throughout its development.

Abstract This chapter consists of five sections outlining aspects related to tools, libraries, formats, and programming languages, as well as considerations that need to be made regarding pre-existing software.

Abstract This chapter first outlines why software and data are relevant to consider when developing safety-critical applications and which requirements need to be considered in this context. We consider both the software that companies buy and the software they develop themselves, but also pay attention to the software already present within the hardware that is in use. The chapter also provides a section on data, pointing out some key aspects relevant to consider in the context of developing AI and safety-critical systems.

Abstract This chapter consists of four sections outlining how to deal with hazards and ongoing risk management. First, we elaborate on how to approach an analysis of a system’s hazards and implement and maintain a hazard log. Next, we elaborate on how ongoing risk management and dynamic risk analysis should ensure that high risk systems are under continuous monitoring and regular review. In the process of hazard identification and risk management, having clearly formulated risk tolerability criteria is paramount. Therefore, Sect. 12.3 outlines several approaches one could use for this purpose. When addressing a high risk system’s possible hazards and risk management strategies, it is important to also ensure the effective management and integration of the specific rules, limitations, and constraints that must be followed and monitored when using the system. The last section therefore outlines on safety-related application conditions (SRAC).

Abstract This chapter first provides a section on artificial intelligence (AI) in high risk systems, giving an overview over the current progress in standards relating to this topic. Next, a section addresses explainable AI (XAI) both as a technical concept and as a concept that has evident human and organizational sides to it. Lastly, a section on the concept of safety of intended functionality (SOTIF) is provided as it addresses safety in AI-driven systems, especially autonomous vehicles. This approach helps mitigate risks from functional insufficiencies in AI algorithms, making it vital for deploying AI safely in high risk areas.

The research presented in this paper highlights the need for updating and expanding the chapters and content of safety cases to accommodate the evolving technological and regulatory landscape, particularly with the integration of AI in safety-critical systems. We have evaluated the relevant topics and chapters for a safety case based on studying relevant safety standards, safety regulations, the EU AI Act, current literature, and interviews with safety experts. As a basis, we have used EN 50129 for railway, which divides safety cases into six parts. We suggest four new parts being included. We included an introduction part (0) to establish scope and context since our solution includes several domains. Due to recent developments in AI, AI regulations and AI standards (RQ1) we have established two new parts: Part 7 “The AI safety report” and Part 8 “Human aspects and oversight”. To ensure that our safety case solution can be used in different domains (RQ2), we have also established two new parts: Part 2, “Operational design domain,” and Part 3, “Concept of operation.” In addition, our AI and cross-domain evaluations resulted in several new and improved subchapters, as shown in the table below. We elaborate on the content of the suggested new parts and which subchapters require improvements.

Abstract Human considerations are central to the development of AI systems, especially in the context of agile safety planning. This chapter explores essential human aspects, such as situational awareness, controllability, and human oversight, which play a critical role in ensuring that high-risk AI systems operate safely and effectively. Through principles of human-centred design, the chapter also examines how interface design, agile development methods, and safety standards can support effective human–machine interaction. This understanding is essential for building AI systems that responsibly balance human involvement with automated processes in safety-critical environments.

Abstract This chapter addresses stakeholders and organizations by first outlining different stakeholder groups to consider in the context of high risk AI systems. Next, it discusses relevant organizational aspects and underscores the importance of safety culture.

Abstract This chapter covers relevant topics related to procurement and subcontractors. First, we outline which aspects developers should consider when buying components to be used as parts of the system being developed. These aspects cover how to organize the procurement process by clearly defining roles and responsibilities, as well as how to ensure access and insights into relevant certificates and documentation accompanying the components. Next, we outline how subcontracting is an important business-related consideration that needs to be addressed in several development projects.

Abstract AI systems are becoming vital in many industries, including safety-critical domains, and are expected to become more complex. This book introduces the AI Act (REGULATION (EU) 2024/1689) and its impact on high risk systems, providing a foundation for safety plans. It targets experts, stakeholders, manufacturers, operators, start-ups, and SMEs. The book focuses on functional safety including automotive, railway, and seaborne systems, covering key aspects for safety plan development, including technical, human, and organizational factors. In the introduction of the book topics covering the AI Act’s risk levels, stakeholder obligations, standards, agile methods, and the role of the Agile Safety Plan in creating safety cases are presented.

Abstract This chapter first provides insight into safety requirements and how they originate from the hazard analysis or relevant standards. Next, it outlines how to ensure that a system or product’s safety is compliant with its functional requirements.

Summary & ConclusionsSafety cases are required by several functional safety standards, specifications, and guidelines. Cybersecurity cases have recently been required by ISO/SAE 21424:2021 for automotive and EN TS 50701:2021 for the railway domain. In this paper we discuss cybersecurity cases and suggest using the topics and structure for a cybersecurity case as described in Annex G of EN TS 50701. BSI PAS 1881:2022 requires: "Trialing organizations shall develop and publish a publicly available and accessible version of the safety case". We have already developed a "safety case for the public" [1] to ensure that (1) the public is aware that safety evidence exists, (2) they are aware of relevant safety aspects when they are passengers, and (3) the vehicle’s limitations are described transparently.Trust is a dynamic process that involves initiating and building trust, responding to violations of trust (failures), and trying to rebuild (repair) trust. The building blocks of trust are not limited to the vehicle itself but also include the embedded AI (Artificial Intelligence) and its overt function. Trust is a holistic perception of the complete service, technology, and organizations responsible for developing, implementing, and certifying an autonomous vehicle.An autonomous vehicle will need acceptance from the certification bodies and the authorities, but we also need to gain the public’s trust. Our research found that several aspects are missing in the safety and cybersecurity cases to ensure public trust.To make self-driving buses a success, they need to be considered trustworthy. Thus, we need a "Trust case" that includes evidence related to distinct trust aspects. Our literature studies, focus groups [4], and surveys found that trust and safety are not correlated. We have developed a "Trust case" to cover the factors not included in the safety and cybersecurity cases. The resulting "Trust case" approach is currently in the form of specific information topics presented in a layman form and a safety case for the public [6], and specific trust topics in [7]. Further research is necessary, related to topics such as deep learning, security, and incorrect reporting to the driver due to e.g., false positive results.

The TrustMe project develops a safety case for autonomous buses.A safety case is mostly based on information from the developers and refers to one or more relevant safety standards.The bases for a safety case are the defined safety standards and proof of compliance, based on the paper trails left by each required activity.A trust case is different and trust and safety assessments are not necessarily correlated.In order to make self-driving buses a success they need to be considered trustworthy.Thus, we need a "Trust case".To ensure that the vehicles are safe and to inform the public, we have developed both a developer safety case and safety case for the public.To take care of the remaining factors we have developed a "Trust case".The trust case has been developed as part of literature studies, surveys and interviews.We have made a survey of 311 passengers and interviewed 18 autonomous bus passengers.Based on literature studies, surveys and interviews, we have proposed a set of issues that should be included into a "Trust case".By providing the public with a "Trust case" together with a "safety case for the public" we will help manufacturers of autonomous vehicles and operators to gain public trust.