The Limitations of Current Similarity-Based Objective Metrics in the Context of Human-Agent Interaction Applications

There are two main ways of evaluating a model generating an interactive virtual agent’s expressions. The first is through subjective perception tests, and the second is through objective metrics, which usually compare the model’s generated expressions to a test set of expressions considered the ground truth. In this work, we argue that using such objective metrics comparing generated expressions, to expressions contained in a test set limits the accuracy of the evaluation by failing to consider expressions that are different from the test set, but are still valid and well-perceived by users. We support this argument through experiments showing that different expression sequences are well perceived as listening responses to the same speaker’s utterance.