DMCA
Automatic evaluation of topic coherence (2010)
Cached
Download Links
| Venue: | In NAACL-HLT |
| Citations: | 85 - 9 self |
BibTeX
@INPROCEEDINGS{Newman10automaticevaluation,
author = {David Newman and Jey Han Lau and Karl Grieser and Timothy Baldwin},
title = {Automatic evaluation of topic coherence},
booktitle = {In NAACL-HLT},
year = {2010}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
This paper introduces the novel task of topic coherence evaluation, whereby a set of words, as generated by a topic model, is rated for coherence or interpretability. We apply a range of topic scoring models to the evaluation task, drawing on WordNet, Wikipedia and the Google search engine, and existing research on lexical similarity/relatedness. In comparison with human scores for a set of learned topics over two distinct datasets, we show a simple co-occurrence measure based on pointwise mutual information over Wikipedia data is able to achieve results for the task at or nearing the level of inter-annotator correlation, and that other Wikipedia-based lexical relatedness methods also achieve strong results. Google produces strong, if less consistent, results, while our results over WordNet are patchy at best. 1
Keyphrases
automatic evaluation topic coherence learned topic google search engine human score evaluation task wikipedia data pointwise mutual information strong result simple co-occurrence measure topic model wikipedia-based lexical relatedness method novel task topic coherence evaluation inter-annotator correlation lexical similarity relatedness distinct datasets
