J. Langford and D. McAllester. Computable shell decomposition bounds. In Proceedings of the International Conference on Computational Learning Theory, 2000.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....strongly from this binomial distributions. Reasonably accurate estimates can be obtained by following a Markov Chain Monte Carlo [4] approach to estimating the prior, using the available database (see Section 4) For an extended discussion of the complexity of estimating this distributions, see [9, 6]. predictive accuracy 0 10 20 30 40 50 60 70 80 support 0 0.2 0.4 0.6 0.8 1 confidence 0 0.1 0.2 0.3 0.4 0.5 0.6 predictive accuracy Fig. 1. Contributions of support s(x) and con dence c( x ) y] to predictive accuracy c( x ) y] of rule [x ) y] Example curve. Figure 1 ....

J. Langford and D. McAllester. Computable shell decomposition bounds. In Proceedings of the International Conference on Computational Learning Theory, 2000.

.... work on algorithms with stochastic guarantees has concentrated on predictive learning with instanceaveraging utility functions, and has pursued two approaches either processing a xed amount of data and making the guarantee dependent on the observed empirical utility values (e.g. Freund, 1998; Langford McAllester, 2000), or demanding a certain xed quality and making the number of examples dependent on the observed utility values (Wald, 1947; Maron Moore, 1994; Greiner, 1996; Domingo et al. 1999) this is often referred to as sequential sampling) In this paper, we generalize known sampling results in two ....

....we would like a particular quality guarantee, we can ask how large a sample we need to draw to ensure that guarantee. The former question has been addressed for predictive learning in work on selfbounding learning algorithms (Freund, 1998) and shell decomposition bounds (Haussler et al. 1996; Langford McAllester, 2000). For our purposes here, the latter question is more interesting. We assume that samples can be requested incrementally from an oracle ( incremental learning ) We can then dynamically adjust the required sample size based on the characteristics of the data that have already been seen; this idea ....

Langford, J., & McAllester, D. (2000). Computable shell decomposition bounds. Proceedings of the International Conference on Computational Learning Theory.

....amount of data. One approach to nding practical algorithms is to process a xed amount of data but determine the possible strength of the quality guarantee dynamically, based on characteristics of the data; this is the idea of self bounding learning algorithms [8] and shell decomposition bounds [13, 19]. Another approach (which we pursue) is to demand a certain xed quality and determine the required sample size dynamically based on characteristics of the data that have already been seen; this idea has originally been referred to as sequential analysis [4, 28, 9] In the machine learning ....

....(in which all hypotheses are equally good) Instead of operating with smaller samples, it is also possible to work with a xed size sample but guarantee a higher quality of the solution if the observed situation di ers from this worst case. This is the general idea of shell decomposition bounds [13, 19] and self bounding learning algorithms [8] Although we have discussed our algorithm only in the context of knowledge discovery tasks, it should be noted that the problem which we address is relevant in a much wider context. A learning agent that actively collects data and searches for a ....

J. Langford and D. McAllester. Computable shell decomposition bounds. In Proceedings of the International Conference on Computational Learning Theory, 2000.

....question is whether there are useful con dence bounds for the predictions. Ideally, the predicted error should come with a con dence interval which is based on the sample size and number of hypotheses which were used to estimate the error histogram. One result in this direction was obtained in [3]. A second fundamental question involves handling in nite hypothesis languages and learners that do not break ties at random. Finally, the analysis does not predict the e ect of local changes of hypotheses (e.g. tree pruning) well. Related Work. Average case analyses have been presented for ....

J. Langford and D. McAllester. Computable shell decomposition bounds. In Proceedings of the International Conference on Computational Learning Theory, 2000.

....of a ( jH i ) with a small error into Equation 1 can result in an inaccurate output of (h L i ) from Equation 1. In this theoretical worst case, estimating suciently accurately can be as dicult as running a learning algorithm, see Sche er (1999) for a more detailed discussion. Recently, Langford and McAllester (2000) have shown that this estimate converges to the true density when m grows, even if m grows parallel to jH i j and log jH i j m does not vanish. Replacing the density ( jH i ) by the distribution (ejH i ) also turns all the integrals into sums that can easily be evaluated. 3. Expected Error ....

....the predictions of Equation 1 are not perfectly accurate; the independence assumptions are the only remaining source of this inaccuracy. It would be particularly interesting to remove the assumption of independent holdout errors of distinct cross validation folds. For error minimizing learners, Langford and McAllester (2000) have in fact removed all independence assumptions, but they arrive at a worst case analysis again (that bene ts from the additional information of ( jH i ) It would be interesting to see if there is an ecient average case analysis of cross validation with weaker assumptions. Acknowledgement ....

Langford, J., & McAllester, D. (2000). Computable shell decomposition bounds. Unpublished manuscript, CMU, Pittsburgh.

....of a ( jH i ) with a small error into Equation 1 can result in an inaccurate output of (h L i ) from Equation 1. In this theoretical worst case, estimating suciently accurately can be as dicult as running a learning algorithm; see Sche er (1999) for a more detailed discussion. Recently, Langford and McAllester (2000) have shown that this estimate converges to the true density when m grows, even if m grows parallel to jH i j and log jH i j m does not vanish. Replacing the density ( jH i ) by the distribution (ejH i ) also turns all the integrals into sums that can easily be evaluated. 3. Expected Error ....

....the predictions of Equation 1 are not perfectly accurate; the independence assumptions are the only remaining source of this inaccuracy. It would be particularly interesting to remove the assumption of independent holdout errors of distinct cross validation folds. For error minimizing learners, Langford and McAllester (2000) have in fact removed all independence assumptions, but they arrive at a worst case analysis again (that bene ts from the additional information of ( jH i ) It would be interesting to see if there is an ecient average case analysis of cross validation with weaker assumptions. Acknowledgement ....

Langford, J., & McAllester, D. (2000). Computable shell decomposition bounds. Unpublished manuscript, CMU, Pittsburgh.

....space with the uniform distribution as stationary distribution [4] This raises the question whether estimating the error histogram of a model class suciently accurately is any easier than estimating the error rate of all hypotheses in that model class. Fortunately, Langford and McAllester [8] have answered this question armatively. It is obvious that the empirical error histogram converges toward the true error histogram when m grows in other words, lim m 1 P (ejH i ) jH i ) However, when m goes to in nity, then all empirical error rates converge to their corresponding true ....

....large. Consider a process in which both the sample size m i and the size of the model class grow in parallel when i 1, such that log jH i j m i stays constantly large. Over this process, we are unable to estimate all error rates in H i but P (ejH i ) converges to ( jH i ) as i grows [8]. In this respect, estimating the histogram is much easier than estimating all error rates in H i . For an extended discussion on the complexity and accuracy of estimating , see [14] The objective of the next experiment is to check whether our analysis can predict the error rate of a decision ....

[Article contains additional citation context not shown here]

J. Langford and D. McAllester. Computable shell decomposition bounds. In Proceedings of the International Conference on Computational Learning Theory, 2000.

No context found.

J. Langford and D. McAllester. Computable shell decomposition bounds. In Proceedings of the International Conference on Computational Learning Theory, 2000.

No context found.

J. Langford and D. McAllester. Computable shell decomposition bounds. In Proceedings of the International Conference on Computational Learning Theory, 2000.

No context found.

J. Langford and D. McAllester. Computable shell decomposition bounds. In Proceedings of the International Conference on Computational Learning Theory, 2000.

No context found.

J. Langford and D. McAllester. Computable shell decomposition bounds. In Proceedings of the International Conference on Computational Learning Theory, 2000.

No context found.

J. Langford and D. McAllester. Computable shell decomposition bounds. In Proceedings of the International Conference on Computational Learning Theory, 2000.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC