The use of Artificial Intelligence (AI) in high-risk, decision-making scenarios presents technical, safety, and normative challenges; problems that may only be ameliorated by human oversight. However, notions of human oversight lack a common foundational understanding: oversight architectures are not well defined, the roles involved remain unclear, and implementation steps are opaque. Hence, researchers and practitioners struggle to determine how to design, implement, and evaluate systems that enable effective human oversight. This paper advances a practical framework for effective human oversight of AI systems, based on a cross-disciplinary perspective that draws on insights from computer science, human-computer interaction, psychology, philosophy, and law. The core contributions are: (1) a foundational framework, with a working definition, architecture and processes for effective human oversight of AI systems; (2) an initial template for documenting oversight architectures and processes, applied to diverse domains; and (3) a synthesis of open research challenges that need to be considered in the emerging field of effective human oversight of AI systems.

We consider the effectiveness of multiobjective counterfactual explanations (MOCEs) in helping individuals learn tactics, or rules of thumb, to apply when required to select a course of action in a specific context. In this setting, a counterfactual explanation compares one course of action against another. A MOCE presents this comparison by highlighting how the two options differ across a range of objectives or metrics. We conduct a study in which participants are presented with various scenarios alongside courses of action that could be implemented in those scenarios. Counterfactual explanations, including those involving multiple objectives, are used to identify the positive and negative aspects of the provided options. Participants were then required to identify the best course of action in various contexts. Participants trained with MOCE outperformed those given no explanations in seven of eight scenarios and those given single-objective counterfactual explanations (SOCEs) in four. SOCEs gave participants an aggregated outcome (expected rewards) without breaking these into specific objectives. MOCE improved tactic learning, but participants provided with SOCE or no explanation performed better in multitactic scenarios. These findings suggest that MOCE enhances tactical decision-making, but further research is needed for multitactic integration.

In this article, we introduce and evaluate a tool for researchers and practitioners to assess the actionability of information provided to users to support algorithmic recourse. While there are clear benefits of recourse from the user’s perspective, the notion of actionability in explainable AI research remains vague, and claims of ‘actionable’ explainability techniques are based on researchers’ intuitions. Inspired by definitions and instruments for assessing actionability in other domains, we construct a seven-item tool and investigate its effectiveness through two user studies. We show that the tool discriminates actionability across explanation types and that the distinctions align with human judgments. We illustrate the impact of context on actionability assessments, suggesting that domain-specific tool adaptations may foster more human-centred algorithmic systems. This is a valuable step forward for research and practices into actionable explainability and algorithmic recourse, providing the first clear human-centred tool for assessing actionability in explainable AI.

The AAMAS conference was established in 2002 as the merger of three highly successful conferences: AA (the International Conference on Autonomous Agents), ICMAS (the International Conference on Multiagent Systems) and ATAL (the International Workshop on Agent Theories, Architectures, and Languages). In this talk I draw on my own experiences and that of others in investigating human-agent collectives. I reflect on aspects of the trajectory of research topics in the AAMAS community over the past 20+ years, and on selected challenges for human-centred AI.

In this paper, we introduce and evaluate a tool for researchers and practitioners to assess the actionability of information provided to users to support algorithmic recourse. While there are clear benefits of recourse from the user's perspective, the notion of actionability in explainable AI research remains vague, and claims of `actionable' explainability techniques are based on the researchers' intuition. Inspired by definitions and instruments for assessing actionability in other domains, we construct a seven-question tool and evaluate its effectiveness through two user studies. We show that the tool discriminates actionability across explanation types and that the distinctions align with human judgements. We show the impact of context on actionability assessments, suggesting that domain-specific tool adaptations may foster more human-centred algorithmic systems. This is a significant step forward for research and practices into actionable explainability and algorithmic recourse, providing the first clear human-centred definition and tool for assessing actionability in explainable AI.

This paper presents Visual Evaluative AI, a decision aid that provides positive and negative evidence from image data for a given hypothesis. This tool finds high-level human concepts in an image and generates the Weight of Evidence (WoE) for each hypothesis in the decision-making process. We apply and evaluate this tool in the skin cancer domain by building a web-based application that allows users to upload a dermatoscopic image, select a hypothesis and analyse their decisions by evaluating the provided evidence. Further, we demonstrate the effectiveness of Visual Evaluative AI on different concept-based explanation approaches.

Prior research on AI-assisted human decision-making has explored several different explainable AI (XAI) approaches. A recent paper has proposed a paradigm shift calling for hypothesis-driven XAI through a conceptual framework called evaluative AI that gives people evidence that supports or refutes hypotheses without necessarily giving a decision-aid recommendation. In this paper, we describe and evaluate an approach for hypothesis-driven XAI based on the Weight of Evidence (WoE) framework, which generates both positive and negative evidence for a given hypothesis. Through human behavioural experiments, we show that our hypothesis-driven approach increases decision accuracy and reduces reliance compared to a recommendation-driven approach and an AI-explanation-only baseline, but with a small increase in under-reliance compared to the recommendation-driven approach. Further, we show that participants used our hypothesis-driven approach in a materially different way to the two baselines.

Research in cognitive psychology has established that whether people prefer simpler explanations to complex ones is context dependent, but the question of `simple vs. complex' becomes critical when an artificial agent seeks to explain its decisions or predictions to humans. We present a model for abstracting causal reasoning chains for the purpose of explanation. This model uses a set of rules to progressively abstract different types of causal information in causal proof traces. We perform online studies using 123 Amazon MTurk participants and with five industry experts over two domains: maritime patrol and weather prediction. We found participants' satisfaction with generated explanations was based on the consistency of relationships among the causes (coherence) that explain an event; and that the important question is not whether people prefer simple or complex explanations, but what types of causal information are relevant to individuals in specific contexts.

In this paper we explore the effect of explanations on reducing errors in the human decision making process caused by placing excessive reliance on automated decision support systems. We develop and implement different forms of explanations based on cognitive principles and evaluate their effect over two different domains: our new version of the Coloured Trails game, and over a simulated radiological task. We found that explanations did not reduce this aspect of automation bias and sometimes increased it. However, they reduced completion time and often increased user decision accuracy, despite not altering the perceived task load. Overall, explanations were beneficial though the benefits were highly context dependent. This work contributes to the complex interplay between automation bias, performance and explanations.

Since the early days of artificial intelligence (AI), many logics have been explored as tools for knowledge representation and reasoning. In the spirit of the Crossley Festscrift and recognizing John Crossley’s diverse interests and his legacy in both mathematical logic and computer science, I discuss examples from my own research that sit in the overlap of logic and AI, with a focus on supporting human–AI interactions.

In this article, we show that explanations of decisions made by machine learning systems can be improved by not only explaining why a decision was made but also explaining how an individual could obtain their desired outcome. We formally define the concept of directive explanations (those that offer specific actions an individual could take to achieve their desired outcome), introduce two forms of directive explanations (directive-specific and directive-generic), and describe how these can be generated computationally. We investigate people’s preference for and perception toward directive explanations through two online studies, one quantitative and the other qualitative, each covering two domains (the credit scoring domain and the employee satisfaction domain). We find a significant preference for both forms of directive explanations compared to non-directive counterfactual explanations. However, we also find that preferences are affected by many aspects, including individual preferences and social factors. We conclude that deciding what type of explanation to provide requires information about the recipients and other contextual information. This reinforces the need for a human-centered and context-specific approach to explainable AI.

Displaying confidence scores in human-AI interaction has been shown to help build trust between humans and AI systems. However, most existing research uses only the confidence score as a form of communication. As confidence scores are just another model output, users may want to understand why the algorithm is confident to determine whether to accept the confidence score. In this paper, we show that counterfactual explanations of confidence scores help study participants to better understand and better trust a machine learning model's prediction. We present two methods for understanding model confidence using counterfactual explanation: (1) based on counterfactual examples; and (2) based on visualisation of the counterfactual space. Both increase understanding and trust for study participants over a baseline of no explanation, but qualitative results show that they are used quite differently, leading to recommendations of when to use each one and directions of designing better explanations.

Many AI applications involve the interaction of multiple autonomous agents, requiring those agents to reason about their own beliefs, as well as those of other agents. However, planning involving nested beliefs is known to be computationally challenging. In this work, we address the task of synthesizing plans that necessitate reasoning about the beliefs of other agents. We plan from the perspective of a single agent with the potential for goals and actions that involve nested beliefs, non-homogeneous agents, co-present observations, and the ability for one agent to reason as if it were another. We formally characterize our notion of planning with nested belief, and subsequently demonstrate how to automatically convert such problems into problems that appeal to classical planning technology for solving efficiently. Our approach represents an important step towards applying the well-established field of automated planning to the challenging task of planning involving nested beliefs of multiple agents.

In this paper, we show that counterfactual explanations of confidence scores help users better understand and better trust an AI model's prediction in human-subject studies. Showing confidence scores in human-agent interaction systems can help build trust between humans and AI systems. However, most existing research only used the confidence score as a form of communication, and we still lack ways to explain why the algorithm is confident. This paper also presents two methods for understanding model confidence using counterfactual explanation: (1) based on counterfactual examples; and (2) based on visualisation of the counterfactual space.

Drawing on an ecological perspective, we contend that research into deception in AI needs to consider not only the cognitive structures of would-be deceptive agents but also the nature of the environments in which they act. To illustrate this approach, we report work-in-progress to design a game called MindTrails, played between a software agent and a human opponent, that is informed by the principles of stage magic to embed deceptive possibilities into its game world. MindTrails is intended to have well-defined elements and rules, while being complex enough to afford a rich range of deceptive behaviours. In this way, it allows us to more precisely articulate some of the deceptive principles of the stage magician and render them more accessible to AI methods and researchers.

In most multiagent applications, communication is essential among agents to coordinate their actions and achieve their goals. However, communication often has a related cost that affects overall system performance. In this paper, we draw inspiration from epistemic planning studies to develop a communication model for agents that allows them to cooperate and make communication decisions effectively within a planning task. The proposed model treats a communication process as an action that modifies the epistemic state of the team. We evaluate whether agents can cooperate effectively and achieve higher performance using communication protocol modeled in our epistemic planning framework in two simulated tasks. Based on an empirical study conducted using search and rescue tasks with different scenarios, our results show that the proposed model improved team performance across all scenarios than baseline models.

We address the challenge of multi-agent system (MAS) design for organisations of agents acting in dynamic and uncertain environments where runtime flexibility is required to enable improvisation through sharing knowledge and adapting behaviour. We identify behavioural features that correspond to runtime improvisation by agents in a MAS organisation and from this analysis describe the OJAzzIC meta-model and an associated design method. We present results from simulation scenarios, varying both problem complexity and the level of organisational support provided in the design, to show that increasing design time guidance in the organisation specification can enable runtime flexibility afforded to agents and improve performance. Hence the results demonstrate the usefulness of the constructs captured in the OJAzzIC meta-model.

Intention recognition is the process of using behavioural cues, such as deliberative actions, eye gaze, and gestures, to infer an agent's goals or future behaviour. In artificial intelligence, one approach for intention recognition is to use a model of possible behaviour to rate intentions as more likely if they are a better ‘fit’ to actions observed so far. In this paper, we draw from literature linking gaze and visual attention, and we propose a novel model of online human intention recognition that combines gaze and model-based AI planning to build probability distributions over a set of possible intentions. In human-behavioural experiments (n=40) involving a multi-player board game, we demonstrate that adding gaze-based priors to model-based intention recognition improved the accuracy of intention recognition by 22% (p<0.05), determined those intentions ≈90 seconds earlier (p<0.05), and at no additional computational cost. We also demonstrate that, when evaluated in the presence of semi-rational or deceptive gaze behaviours, the proposed model is significantly more accurate (9% improvement) (p<0.05) compared to a model-based or gaze only approaches. Our results indicate that the proposed model could be used to design novel human-agent interactions in cases when we are unsure whether a person is honest, deceitful, or semi-rational.