DMCA
On Reliable Crowdsourcing and the Use of Ground Truth Information
BibTeX
@MISC{Liu_onreliable,
author = {Qiang Liu and Mark Steyvers and John W. Fisher and Alexander Ihler},
title = {On Reliable Crowdsourcing and the Use of Ground Truth Information},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Using some expert labels or control questions with known answers may significantly improve the reliabil-ities of crowdsourcing systems, but with high cost that may not be feasible at large scale. This raises important problems on how to optimally integrate such additional, expensive but accurate information with the cheap but unreliable crowd labels, addressing which is critical to build more reliable and scalable crowdsourcing systems in the future. We give an overview of some related open problems, along with some current work. The crowdsourcing paradigm has emerged as a powerful approach for collecting human knowledge and intelligence at large scale. One of its underlying philosophies is the wis-dom of crowds phenomenon (Surowiecki, 2005): properly
