Efficient Online Scalar Annotation with Bounded Support
meeting of the association for computational linguistics, Volume abs/1806.01170, 2018.
We describe a novel method for efficiently eliciting scalar annotations for dataset construction and system quality estimation by human judgments. We contrast direct assessment (annotators assign scores to items directly), online pairwise ranking aggregation (scores derive from annotator comparison of items), and a hybrid approach (EASL: ...More
PPT (Upload PPT)