Trusting Artificial Agents: Communication Trumps Performance

Acceptability and trust toward an Artificial Agent (AA) are known to be strongly related to the transparency of its behavior. However, the opacity of the AI techniques implemented to drive the behavior of AAs is growing in parallel with their performance (performance/transparency trade-off). Thus, it is crucial and increasingly required to identify minimal and necessary information for achieving efficient human/AA interaction in order to include them as AAs' requirements at design stage. For this purpose, this paper proposes to bring knowledge and methods from domains accustomed to human behavior studies. Based on ergonomic and cognition literature, this paper tests through a user-study the hypothesis that sharing distal, proximal and motor intentions (what we call intention-based explanations) will improve the acceptability of an AA. The "Overcooked" task from Carroll and colleagues[3] - which requires coordination on goals and motor levels - is used as a test-bed for hypotheses manipulation. Our experimental work consisted in implementing a modified version of the task, analyzing 60 subjects performance, behaviors and feelings in two groups (control and hypothesis-testing) and having them filled an extensive survey. Half of them interact with an agent sharing its intentions while the other half stand in the control group without any information shared by the agent. The results show that intentions sharing leads to a greater acceptability - by means of delegation of control towards the AA - as well as trust. Critically, acceptability and trust seem to be decoupled from team performance. These results suggest the importance of intention-based explanations as a support for cooperation between the human operator and artificial agents. This work demonstrates the need to take into account human cognition when designing systems requiring acceptable and trustworthy AI techniques.