Do you Feel Certain about your Annotation? A Web-based Semantic Frame Annotation Tool Considering Annotators' Concerns and Behaviors.

In this paper, we present an open-source web-based application with a responsive design for modular semantic frame annotation (SFA). Besides letting experienced and inexperienced users perform suggestion-based and slightly-controlled annotations, the system keeps track of the time and changes annotators made during the annotation process and logs certain metadata. This collected metadata can be used to get new insights regarding the difficulty of annotating specific types of frames, and as an input of an annotation cost measurement for an active learning algorithm. The tool was already used to build a manually annotated corpus with semantic frames and its arguments for task 2 of SemEval 2019 regarding unsupervised lexical frame induction (QasemiZadeh et al., 2019). Although English sentences from the Wall Street Journal corpus of the Penn Treebank (Marcus et al., 1999) are annotated for this task, it is also possible to use the proposed tool for the annotation of sentences in any other languages.