DMCA
Estimating the number of clusters in a dataset via the Gap statistic (2000)
| Citations: | 501 - 1 self |
BibTeX
@ARTICLE{Tibshirani00estimatingthe,
author = {Robert Tibshirani and Guenther Walther and Trevor Hastie},
title = {Estimating the number of clusters in a dataset via the Gap statistic},
journal = {},
year = {2000},
volume = {63},
pages = {411--423}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
We propose a method (the \Gap statistic") for estimating the number of clusters (groups) in a set of data. The technique uses the output of any clustering algorithm (e.g. k-means or hierarchical), comparing the change in within cluster dispersion to that expected under an appropriate reference null distribution. Some theory is developed for the proposal and a simulation study that shows that the Gap statistic usually outperforms other methods that have been proposed in the literature. We also briey explore application of the same technique to the problem for estimating the number of linear principal components. 1 Introduction Cluster analysis is an important tool for \unsupervised" learning| the problem of nding groups in data without the help of a response variable. A major challenge in cluster analysis is estimation of the optimal number of \clusters". Figure 1 (top right) shows a typical plot of an error measure W k (the within cluster dispersion dened below) for a clustering pr...
Keyphrases
gap statistic cluster dispersion typical plot simulation study clustering pr linear principal component appropriate reference null distribution error measure important tool introduction cluster analysis explore application optimal number cluster analysis top right response variable major challenge
