Supervised Learning from Multiple Experts: Whom to trust when everyone lies a bit
by
Vikas C. Raykar
,
Shipeng Yu
,
Linda H. Zhao
,
Anna Jerebko
,
Charles Florin
,
Gerardo Hermosillo Valadez
,
Luca Bogoni
,
Linda Moy
| Citations: | 15 - 2 self |
BibTeX
@MISC{Raykar_supervisedlearning,
author = {Vikas C. Raykar and Shipeng Yu and Linda H. Zhao and Anna Jerebko and Charles Florin and Gerardo Hermosillo Valadez and Luca Bogoni and Linda Moy},
title = {Supervised Learning from Multiple Experts: Whom to trust when everyone lies a bit},
year = {}
}
Bookmark
 |
 |
 |
 |
 |
 |
OpenURL
Abstract
We describe a probabilistic approach for supervised learning when we have multiple experts/annotators providing (possibly noisy) labels but no absolute gold standard. The proposed algorithm evaluates the different experts and also gives an estimate of the actual hidden labels. Experimental results indicate that the proposed method is superior to the commonly used majority voting baseline. 1.
Citations
| 6231 | Maximum likelihood from incomplete data via the EM algorithm - Dempster, Laird, et al. - 1977 |
| 612 | A view of the EM algorithm that justifies incremental, sparse, and other variants - Neal, Hinton - 1998 |
| 90 | Cheap and fast – but is it good? Evaluating nonexpert annotations for natural language tasks - Snow, O’Conner, et al. - 2008 |
| 65 | Get another label? Improving data quality and data mining using multiple, noisy labelers - Sheng, Provost, et al. - 2008 |
| 35 | M.: A Simple Approach to Ordinal Classification - Frank, Hall - 2001 |
| 29 | Inferring ground truth from subjective labelling of Venus images - Smyth, Fayyad, et al. - 1994 |
| 15 | Learning with an unreliable teacher - Lugosi - 1992 |
| 6 | Evaluation of diagnostic tests without gold standards - Hui, Zhou - 1998 |
| 4 | Maximum likeihood estimation of observer error-rates using the EM algorithm - Dawid, Skene - 1979 |
| 4 | Learning with probabilistic supervision,” Computational Learning Theory and - Smyth - 1995 |
