Textual data used to train large language models (LLMs) exhibits multifaceted bias manifestations, encompassing harmful language and skewed demographic distributions. Regulations such as the EU AI Act require identifying and mitigating biases against protected groups in data, with the ultimate goal of preventing unfair model outputs. However, practical guidance and operationalization are lacking. We propose a comprehensive data bias detection and mitigation pipeline comprising four components that address two data bias types, namely representation bias and (explicit) stereotypes for a configurable sensitive attribute. First, we leverage LLM-generated word lists, created according to defined quality criteria, to detect relevant group labels. Second, representation bias is quantified using the Demographic Representation Score. Third, we detect and mitigate stereotypes using sociolinguistically informed filtering. Finally, we mitigate representation bias through Grammar-and Context-Aware Counterfactual Data Augmentation. We conduct a twofold evaluation using gender, religion, and age as examples. First, we evaluate the effectiveness of each individual component on data debiasing through human validation and baseline comparison. The findings demonstrate that we successfully reduce representation bias and (explicit) stereotypes in a text dataset. Second, we evaluate the effect of data debiasing on model bias by benchmarking several models (0.6B-8B parameters) fine-tuned on the debiased text dataset. This evaluation reveals that LLMs fine-tuned on debiased data do not consistently show improved performance on bias benchmarks. These results expose critical gaps in current evaluation methodologies and highlight the need for targeted data interventions to address manifested model bias.

While independent and reproducible software testing is widely established in safety-critical systems and also supported by development and testing infrastructures, there is still no adequate counterpart for testing of AI systems. In contrast, current AI tests tend to be tightly integrated into the development framework and are not modular in the sense that testing code and system-under-test (SUT) are strictly separable in terms of their software environments. In this paper, we present an AI testing framework for trustworthy AI that aims to support independent, reproducible and auditable AI testing by providing a design-pattern of computational testing workflows, which strongly promotes that individual tests are modular, reproducible, and automatable while maintaining a high-degree of auditablility. To demonstrate the viability and usefulness of this framework, we use it to create a workflow template for the case of metric-based testing of AI models using test datasets and implement a proof-of-concept (PoC) for the specific case of performance tests of visual object detectors. This PoC is publicly available on the AI on demand platform (Demo and code accessible from https://bit.ly/4meYnNo).

Studying systematic weaknesses of DNNs has gained prominence in the last few years with the rising focus on building safe AI systems. Slice discovery methods (SDMs) are prominent algorithmic approaches for finding such systematic weaknesses. They identify top-k semantically coherent slices/subsets of data where a DNN-under-test has low performance. For being directly useful, e.g., as evidences in a safety argumentation, slices should be aligned with human-understandable (safety-relevant) dimensions, which, for example, are defined by safety and domain experts as parts of the operational design domain (ODD). While straightforward for structured data, the lack of semantic metadata makes these investigations challenging for unstructured data. Therefore, we propose a complete workflow which combines contemporary foundation models with algorithms for combinatorial search that consider structured data and DNN errors for finding systematic weaknesses in images. In contrast to existing approaches, ours identifies weak slices that are in line with predefined human-understandable dimensions. As the workflow includes foundation models, its intermediate and final results may not always be exact. Therefore, we build into our workflow an approach to address the impact of noisy metadata. We evaluate our approach w.r.t. its quality on four popular computer vision datasets, including autonomous driving datasets like Cityscapes, BDD100k, and RailSem19, while using multiple state-of-the-art models as DNNs-under-test.

Due to broad deployment of AI systems in risk-prone domains and AI regulations coming into effect, systematic risk-and quality assessments of AI systems have become increasingly important. Conducting such assessments involves identifying relevant quality criteria for a given AI system and selecting test methods, i.e., procedures for collecting and evaluating evidences and measurable quantities that fit the identified criteria. This selection process can be challenging due to the high complexity of the test method landscape and, in the context of independent assessments, due to potential conflicts of interest between the involved stakeholders. To address this challenge, we present a practical solution approach for systematic, taxonomy-based selection of test methods. The paper closes with an outline of the gaps and possible next steps for utilizing test methods to achieve scalable and comparable, independent AI assessments.

Social categories and stereotypes are embedded in language and can introduce data bias into Large Language Models (LLMs). Despite safeguards, these biases often persist in model behavior, potentially leading to representational harm in outputs. While sociolinguistic research provides valuable insights into the formation of stereotypes, NLP approaches for stereotype detection rarely draw on this foundation and often lack objectivity, precision, and interpretability. To fill this gap, in this work we propose a new approach that detects and quantifies the linguistic indicators of stereotypes in a sentence. We derive linguistic indicators from the Social Category and Stereotype Communication (SCSC) framework which indicate strong social category formulation and stereotyping in language, and use them to build a categorization scheme. To automate this approach, we instruct different LLMs using in-context learning to apply the approach to a sentence, where the LLM examines the linguistic properties and provides a basis for a fine-grained assessment. Based on an empirical evaluation of the importance of different linguistic indicators, we learn a scoring function that measures the linguistic indicators of a stereotype. Our annotations of stereotyped sentences show that these indicators are present in these sentences and explain the strength of a stereotype. In terms of model performance, our results show that the models generally perform well in detecting and classifying linguistic indicators of category labels used to denote a category, but sometimes struggle to correctly evaluate the associated behaviors and characteristics. Using more few-shot examples within the prompts, significantly improves performance. Model performance increases with size, as Llama-3.3-70B-Instruct and GPT-4 achieve comparable results that surpass those of Mixtral-8x7B-Instruct, GPT-4-mini and Llama-3.1-8B-Instruct.

Bias in training data can lead to algorithmic unfairness in machine learning tasks. Therefore, a general requirement for trustworthy AI is that data should be representative and free of bias. There are several approaches to measure fairness of a given dataset based on attributes such as gender or race. However, for unstructured data, such measures require the dataset to be labeled with respect to these attributes, and cannot be directly applied to unlabeled image datasets. We present an approach using foundation models to analyze the fairness of unlabeled images, exploiting the fact that foundation models implement a semantically consistent mapping from the unstructured image space to the embedding space. In particular, we systematically compare the embedding of a reference dataset known to be “fair” to an unlabeled image dataset. We show that the resulting data structures in the embedding support a systematic comparative analysis based on both qualitative as well as quantitative evaluation. We evaluate our approach analyzing the fairness of the target image dataset CelebA while using the FairFace dataset as reference. The validation against the ground truth labels of the CelebA dataset demonstrates the principal applicability of the overall approach. In sum, our work offers a novel perspective on fairness evaluation of images, as it requires no labeling but rather makes use of existing already labeled reference datasets.

With the advancement of DNNs into safety-critical applications, testing approaches for such models have gained more attention. A current direction is the search for and identification of systematic weaknesses that put safety assumptions based on average performance values at risk. Such weaknesses can take on the form of (semantically coherent) subsets or areas in the input space where a DNN performs systematically worse than its expected average. However, it is non-trivial to attribute the reason for such observed low performances to the specific semantic features that describe the subset. For instance, inhomogeneities within the data w.r.t. other (non-considered) attributes might distort results. However, taking into account all (available) attributes and their interaction is often computationally highly expensive. Inspired by counterfactual explanations, we propose an effective and computationally cheap algorithm to validate the semantic attribution of existing subsets, i.e., to check whether the identified attribute is likely to have caused the degraded performance. We demonstrate this approach on an example from the autonomous driving domain using highly annotated simulated data, where we show for a semantic segmentation model that (i) performance differences among the different pedestrian assets exist, but (ii) only in some cases is the asset type itself the reason for this reduction in the performance.

AI applications bear inherent risks in various risk dimensions, such as insufficient reliability, robustness, fairness or data protection. It is well-known that trade-offs between these dimensions can arise, for example, a highly accurate AI application may reflect unfairness and bias of the real-world data, or may provide hard-to-explain outcomes because of its internal complexity. AI risk assessment frameworks aim to provide systematic approaches to risk assessment in various dimensions. The overall trustworthiness assessment is then generated by some form of risk aggregation among the risk dimensions. This paper provides a systematic overview on risk aggregation schemes used in existing AI risk assessment frameworks, focusing on the question how potential trade-offs among the risk dimensions are incorporated. To this end, we examine how the general risk notion, the application context, the extent of risk quantification, and specific instructions for evaluation may influence overall risk aggregation. We discuss our findings in the current frameworks in terms of whether they provide meaningful and practicable guidance. Lastly, we derive recommendations for the further operationalization of risk aggregation both from horizontal and vertical perspectives.

The trustworthiness of AI applications has been the subject of recent research and is also addressed in the EU's recently adopted AI Regulation. The currently emerging foundation models in the field of text, speech and image processing offer completely new possibilities for developing AI applications. This whitepaper shows how the trustworthiness of an AI application developed with foundation models can be evaluated and ensured. For this purpose, the application-specific, risk-based approach for testing and ensuring the trustworthiness of AI applications, as developed in the 'AI Assessment Catalog - Guideline for Trustworthy Artificial Intelligence' by Fraunhofer IAIS, is transferred to the context of foundation models. Special consideration is given to the fact that specific risks of foundation models can have an impact on the AI application and must also be taken into account when checking trustworthiness. Chapter 1 of the white paper explains the fundamental relationship between foundation models and AI applications based on them in terms of trustworthiness. Chapter 2 provides an introduction to the technical construction of foundation models and Chapter 3 shows how AI applications can be developed based on them. Chapter 4 provides an overview of the resulting risks regarding trustworthiness. Chapter 5 shows which requirements for AI applications and foundation models are to be expected according to the draft of the European Union's AI Regulation and Chapter 6 finally shows the system and procedure for meeting trustworthiness requirements.

As AI applications are emerging in diverse fields - e.g., industry, healthcare or finance - weaknesses and failures of such applications might bare unacceptable risks which need to be rigorously assessed, quantified and, if necessary, mitigated. One crucial component of an effective AI trustworthiness assessment and risk management are systematic evaluations of the AI application based on properly chosen and executed tests. In addition to the known requirements of providing facilities for automated and reproducible tests, an assessment platform for Trustworthy AI must support the integration of different AI models and data sets, must be extensible for AI risk specific metrics and test tools, and should facilitate collaboration between model providers, assessment tool developers and auditors. In this paper, we develop an architecture of a platform for automated, reproducible and collaborative assessments of AI applications, based on an in-depth requirements analysis that maps use cases and collaboration scenarios to technical requirements.

With the current interest in deploying machine learning (ML) models in safety-critical applications like automated driving (AD), there is increased effort in developing sophisticated testing techniques for evaluating the models. One of the primary requirements for testing is the availability of test data, particularly test data that captures the long tail distributions of traffic events. As such data collection in the real world is hazardous, there is also a necessity for generating synthetic data using simulators or deep learning-based approaches. We propose a pipeline to generate augmented safety-critical scenes of the Cityscapes dataset using pre-trained SOTA latent diffusion models with additional conditioning using text and OpenPose-based ControlNet, where we have fine-grained control of the attributes of the generated pedestrians. In addition, we propose a filtering mechanism, similar to self-consistency checks in large language models (LLMs), to improve the quality of the generated data regarding the adherence to generated attributes, reaching similar to 25% improvement in our experiments. Finally, using pre-trained SOTA segmentation models on Cityscapes, we evaluate the generated dataset's viability by qualitatively evaluating the predicted segmentation maps.

Artificial Intelligence (AI) has made impressive progress in recent years and represents a key technology that has a crucial impact on the economy and society. However, it is clear that AI and business models based on it can only reach their full potential if AI applications are developed according to high quality standards and are effectively protected against new AI risks. For instance, AI bears the risk of unfair treatment of individuals when processing personal data e.g., to support credit lending or staff recruitment decisions. The emergence of these new risks is closely linked to the fact that the behavior of AI applications, particularly those based on Machine Learning (ML), is essentially learned from large volumes of data and is not predetermined by fixed programmed rules. Thus, the issue of the trustworthiness of AI applications is crucial and is the subject of numerous major publications by stakeholders in politics, business and society. In addition, there is mutual agreement that the requirements for trustworthy AI, which are often described in an abstract way, must now be made clear and tangible. One challenge to overcome here relates to the fact that the specific quality criteria for an AI application depend heavily on the application context and possible measures to fulfill them in turn depend heavily on the AI technology used. Lastly, practical assessment procedures are needed to evaluate whether specific AI applications have been developed according to adequate quality standards. This AI assessment catalog addresses exactly this point and is intended for two target groups: Firstly, it provides developers with a guideline for systematically making their AI applications trustworthy. Secondly, it guides assessors and auditors on how to examine AI applications for trustworthiness in a structured way.

Using Machine Learning (ML) models for safety-critical perception tasks in Autonomous Driving (AD) or other domains requires a thorough evaluation of the model performance and the data coverage w.r.t. the intended Operational Design Domain (ODD). However, obtaining the needed per-image semantic meta-data along the relevant dimensions of the ODD for real-world image datasets is non-trivial. Recent advances in self-supervised foundation models, specifically CLIP, suggest that such meta-data could be obtained for real-world images in an automated fashion using zero-shot classification. While CLIP was already reported to achieve promising performance on tasks such as the recognition of gender or age on facial images, we investigate to which extent less prominent and more fine-grained observables, e.g., presence of accessories such as spectacles or the shirt- or hair-color, can be determined. We provide an analysis of CLIP for generating fine-grained meta-data on three datasets from the AD domain, one of synthetic origin including ground truth, the others being Cityscapes and Railsem19. We also compare with a standard facial dataset where more elaborate attribute annotations are present. To improve the quality of generated meta-data, we additionally extend the ensemble approach of CLIP by a simple noise-suppressing technique.

Our Visual Analytics (VA) tool ScrutinAI supports human analysts to investigate interactively model performance and data sets. Model performance depends on labeling quality to a large extent. In particular in medical settings, generation of high quality labels requires in depth expert knowledge and is very costly. Often, data sets are labeled by collecting opinions of groups of experts. We use our VA tool to analyze the influence of label variations between different experts on the model performance. ScrutinAI facilitates to perform a root cause analysis that distinguishes weaknesses of deep neural network (DNN) models caused by varying or missing labeling quality from true weaknesses. We scrutinize the overall detection of intracranial hemorrhages and the more subtle differentiation between subtypes in a publicly available data set.

Artificial Intelligence (AI) has made impressive progress in recent years and represents a key technology that has a crucial impact on the economy and society. However, it is clear that AI and business models based on it can only reach their full potential if AI applications are developed according to high quality standards and are effectively protected against new AI risks. For instance, AI bears the risk of unfair treatment of individuals when processing personal data e.g., to support credit lending or staff recruitment decisions. The emergence of these new risks is closely linked to the fact that the behavior of AI applications, particularly those based on Machine Learning (ML), is essentially learned from large volumes of data and is not predetermined by fixed programmed rules. Thus, the issue of the trustworthiness of AI applications is crucial and is the subject of numerous major publications by stakeholders in politics, business and society. In addition, there is mutual agreement that the requirements for trustworthy AI, which are often described in an abstract way, must now be made clear and tangible. One challenge to overcome here relates to the fact that the specific quality criteria for an AI application depend heavily on the application context and possible measures to fulfill them in turn depend heavily on the AI technology used. Lastly, practical assessment procedures are needed to evaluate whether specific AI applications have been developed according to adequate quality standards. This AI assessment catalog addresses exactly this point and is intended for two target groups: Firstly, it provides developers with a guideline for systematically making their AI applications trustworthy. Secondly, it guides assessors and auditors on how to examine AI applications for trustworthiness in a structured way.

We present ScrutinAI, a Visual Analytics tool to leverage semantic understanding for deep neural network (DNN) prediction analysis, focusing on models for object detection and semantic segmentation. Typical fields of application for such models, e.g., autonomous driving or healthcare, have a high demand for uncovering and mitigating data-and model-inherent shortcomings. Our approach helps analysts and/or auditors use their semantic understanding to identify and investigate potential weaknesses in DNN models. ScrutinAI includes interactive visualizations of the model's inputs and outputs, interactive plots with linked brushing, data filtering with textual queries on descriptive meta data and an interactive similarity based image retrieval feature. The different views and data filtering options enable global and local inspection of DNN predictions. Overall, the tool fosters hypothesis driven knowledge generation which aids in understanding the model's inner reasoning. Insights gained during the analysis process mitigate the "black-box character"of the DNN and thus support model improvement and the generation of a safety argumentation for AI applications. We present two case studies on the investigation of DNN models for pedestrian detection from the automotive domain.& COPY; 2023 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

We present ScrutinAI, a Visual Analytics approach to exploit semantic understanding for deep neural network (DNN) predictions analysis, focusing on models for object detection and semantic segmentation. Typical fields of application for such models, e.g. autonomous driving or healthcare, have a high demand for detecting and mitigating data- and model-inherent shortcom-ings. Our approach aims to help analysts use their semantic understanding to identify and investigate potential weaknesses in DNN models. ScrutinAI therefore includes interactive visualizations of the model’s inputs and outputs, interactive plots with linked brushing, and data filtering with textual queries on descriptive meta data. The tool fosters hypothesis driven knowledge generation which aids in understanding the model’s inner reasoning. Insights gained during the analysis process mitigate the “black-box character” of the DNN and thus support model improvement and generation of a safety argumentation for AI applications. We present a case study on the investigation of DNN models for pedestrian detection from the automotive domain. sun angle or the weather condition, and finer, object-specific details on the other hand, e.g. the gender of a pedestrian and the color of their clothes. As development of ScrutinAI is still work in progress, we present our interim results with a case study in the context of trustworthy AI for autonomous driving.

AbstractThe latest generation of safety standards applicable to automated driving systems require both qualitative and quantitative safety acceptance criteria to be defined and argued. At the same time, the use of machine learning (ML) functions is increasingly seen as a prerequisite to achieving the necessary levels of perception performance in the complex operating environments of these functions. This inevitably leads to the question of which supporting evidence must be presented to demonstrate the safety of ML-based automated driving systems. This chapter discusses the challenge of deriving suitable acceptance criteria for the ML function and describes how such evidence can be structured in order to support a convincing safety assurance case for the system. In particular, we show how a combination of methods can be used to estimate the overall machine learning performance, as well as to evaluate and reduce the impact of ML-specific insufficiencies, both during design and operation.