Characterizing Model Performance in the Feature Space
|
Abstract:
A fundamental problem in machine learning is understanding the conditions for which a learning algorithm works well. Understanding an algorithm's strengths and weaknesses and being able to compare two algorithms with each other are necessary for designers to develop (or select) learning algorithms for a specific problem. Generally, one can attempt to analyze and understand algorithms either theoretically or empirically. Theoretical analyses of machine learning algorithms have usually resulted in weak performance guarantees that are not much use to a practitioner. Algorithms are typically proven to be asymptotically consistent (i.e., will achieve the Bayes optimal error rate given enough training examples) or that the algorithm can be used to PAC (Probably Approximately Correct) learn a given concept (Valiant, 1984). Another approach is to analyze average case behavior under specific distributional assumptions, such as learning m-of-n concepts (Langley & Sage, 1999). Although these analyses are useful in understanding the general behavior of an algorithm, they are unable to provide guidance to the designer in the form of specific predictions of an algorithm's performance with a given problem. Thus most researchers and practitioners resort to empirical evaluation to understand the interaction between learning algorithms and a domain. Unfortunately, most evaluation methods give very little information to the designer. For example, the most common method of empirically evaluating a classifier is to examine its error, or more generally loss,
Citations
| 2138 | UCI Repository of Machine Learning Databases – Merz, Murphy - 1996 |
| 1364 | A theory of the learnable – Valiant - 1984 |
| 129 | Data mining using MLC++: a machine learning library in C – Kohavi, Sommerfield, et al. - 1996 |
| 88 | Generalizing from case studies: a case study – Aha - 1992 |
| 41 | Discriminative vs informative learning – Rubinstein, Hastie - 1997 |
| 30 | Detecting change in categorical data: Mining contrast sets – Bay, Pazzani - 1999 |
| 15 | Dynamical selection of learning algorithms – Merz - 1995 |
| 12 | Characterizing model errors and differences – Bay, Pazzani - 2000 |
| 7 | Tractable average-case analysis of naive bayesian classifiers – Langley, Sage - 1999 |
| 1 | Characterizing model errors and di erences – Bay, &Pazzani - 2000 |
