A Bayesian Framework for Modeling Human EvaluationsEI
Several situations that we come across in our daily lives involve some form of evaluation: a process where an evaluator chooses a correct label for a given item. Examples of such situations include a crowd-worker labeling an image or a student answering a multiple-choice question. Gaining insights into human evaluations is important for determining the quality of individual evaluators as well as identifying true labels of items. Here, we generalize the question of estimating the quality of individual evaluators, extending it to obtain...更多
- 4Charles E. Antoniak, Mixtures of Dirichlet Processes with Applications to Bayesian Nonparametric Problems.Annals of Statistics, pp. 1152-1174, 1974.
- 6Ece Kamar, Severin Hacker, Eric Horvitz. Combining human and machine intelligence in large-scale crowdsourcing.AAMAS, pp. 467-474, 2012.
- 8Chao Liu, Yi-Min Wang. TrueLabel + Confusions: A Spectrum of Probabilistic Models in Analyzing Multiple Ratings.international conference on machine learning, pp. 225-232, 2012.
- 11Dengyong Zhou, John C. Platt, Sumit Basu, Yi Mao. Learning from the Wisdom of Crowds by Minimax Entropy.NIPS, pp. 2204-2212, 2012.
- 12Welinder, P.; Perona, P.. Online crowdsourcing: Rating annotators and obtaining cost-effective labels.Computer Vision and Pattern Recognition Workshops, pp. 25-32, 2010.
SDM, pp. 181-189, 2015.