Citation Context

...intly estimated by maximizing the likelihood of the observed worker labels, where the unobserved true labels are treated as latent variables. Although this EM-based approach has had empirical success =-=[21, 20, 19, 26, 6, 25]-=-, there is as yet no theoretical guarantee for its performance. A recent theoretical study [10] shows that the global optimal solutions of the Dawid-Skene estimator can achieve minimax rates of conver...

Citation Context

...ple times by different workers, then the redundant labels can provide hints on resolving the true labels. Much progress has been made in designing effective aggregation mechanisms to infer the true labels from noisy observations. From a modeling perspective, existing work includes both generative approaches and discriminative approaches. A generative method builds a flexible probabilistic model for generating the noisy observations conditioned on the unknown true labels and some behavior assumptions, with examples of the Dawid-Skene (DS) estimator [5], the minimax entropy (Entropy) estimator1 [24, 25], and their variants. In contrast, a discriminative approach does not model the observations; it directly identifies the true labels via some aggregation rules. Examples include majority voting and the weighted majority voting that takes worker reliability into consideration [10, 11]. In this paper, we present a max-margin formulation of the most popular majority voting estimator to improve its discriminative ability, and further present a Bayesian generalization that conjoins the advantages of both generative and discriminative approaches. The max-margin majority voting (M3V) directly maximiz...

Citation Context

...sourced labels. Recent rapid take-up of crowdsourcing marketplaces, e.g., Amazon MTurk1, provides a cheap way of collecting large supervised datasets (Muhammadi et al., 2013). However, due to a lack of expertise and presence of spammer workers, such labels often hugely vary in quality. To overcome this issue, employers assign each item to multi1http://www.mturk.com/ MGBDT Model for Weight and Target Learning ple workers. Afterwards, item’s multiple labels are aggregated into a consensus label with the use of a certain consensus algorithm, e.g., majority voting, label averaging, etc (see, e.g. Zhou et al., 2014). Various consensus models are known to significantly improve the precision of raw crowd labels. The examples above share one common property, which is crucial for the algorithm we propose in this paper. Namely, for each item in the training dataset, besides a noisy label itself, we observe several additional variables, which altogether are further referred to as label features. Examples of label features are: fact of a click, click position, and dwell time in the click prediction problem; worker’s experience, outputs of various consensus models in the problem of learning with crowdsourced lab...

Citation Context

... the problem of obtaining confidence bounds on the worker error rates, we consider the problem of finding the maximum likelihood estimates to the item ground truth and worker error rates. Zhou et al. =-=[29, 30]-=- use minimax entropy to perform worker quality estimation as well as inherent item difficulty estimation; here the inherent item difficulty is represented as a vector. Their technique only applies whe...