Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collections Accurately and Affordably
DESIRES, pp. 42-46, 2018.
Crowdsourcing offers an affordable and scalable means to collect relevance judgments for IR test collections. However, crowd assessors may show higher variance in judgment quality than trusted assessors. In this paper, we investigate how to effectively utilize both groups of assessors in partnership. We specifically investigate how agreem...More
Full Text (Upload PDF)
PPT (Upload PPT)