Improving Taxonomy of Errors in Chat-Oriented Dialogue Systems

In previous studies, top-down and bottom-up approaches have been proposed for creating taxonomies of errors in chat-oriented dialogue systems. However, the reported \(\kappa \) (kappa) value for the taxonomy based on the top-down approach is low at 0.239, and no evaluation has been conducted for that based on the bottom-up approach. In this paper, we propose to revise these taxonomies to achieve better inter-annotator agreement. The revised taxonomy based on the bottom-up approach yielded a reasonable \(\kappa \) of 0.44 (Fleiss’ \(\kappa \)), suggesting that this taxonomy can be used reliably to classify errors in chat-oriented dialogue systems.