A study of translation edit rate with targeted human annotation

DMCA

A study of translation edit rate with targeted human annotation (2006)

Cached

Download Links

by Matthew Snover , Bonnie Dorr , Richard Schwartz , Linnea Micciulla , John Makhoul

Venue:	In Proceedings of Association for Machine Translation in the Americas
Citations:	583 - 9 self

BibTeX

@INPROCEEDINGS{Snover06astudy,
    author = {Matthew Snover and Bonnie Dorr and Richard Schwartz and Linnea Micciulla and John Makhoul},
    title = {A study of translation edit rate with targeted human annotation},
    booktitle = {In Proceedings of Association for Machine Translation in the Americas},
    year = {2006},
    pages = {223--231}
}

OpenURL

Abstract

We examine a new, intuitive measure for evaluating machine-translation output that avoids the knowledge intensiveness of more meaning-based approaches, and the labor-intensiveness of human judgments. Translation Edit Rate (TER) measures the amount of editing that a human would have to perform to change a system output so it exactly matches a reference translation. We show that the single-reference variant of TER correlates as well with human judgments of MT quality as the four-reference variant of BLEU. We also define a human-targeted TER (or HTER) and show that it yields higher correlations with human judgments than BLEU—even when BLEU is given human-targeted references. Our results indicate that HTER correlates with human judgments better than HMETEOR and that the four-reference variants of TER and HTER correlate with human judgments as well as—or better than—a second human judgment does. 1

Keyphrases

human judgment translation edit rate human annotation four-reference variant meaning-based approach intuitive measure knowledge intensiveness single-reference variant reference translation ter correlate machine-translation output system output second human judgment mt quality human-targeted reference human-targeted ter

DMCA