DMCA
Inter-Coder Agreement for Computational Linguistics (2008)
| Venue: | COMPUTATIONAL LINGUISTICS |
| Citations: | 240 - 7 self |
BibTeX
@ARTICLE{Artstein08inter-coderagreement,
author = {Ron Artstein and Massimo Poesio},
title = {Inter-Coder Agreement for Computational Linguistics},
journal = {COMPUTATIONAL LINGUISTICS},
year = {2008},
volume = {34},
number = {4},
pages = {555--596}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff’s alpha as well as Scott’s pi and Cohen’s kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in Computational Linguistics, may be more appropriate for many corpus annotation tasks – but that their use makes the interpretation of the value of the coefficient even harder.
