Kearns, M.: E#cient noise-tolerant learning from statistical queries. Journal of the ACM 45 (1998) 983--1006  Home/Search   Document Not in Database   Summary   Related Articles   Check

.... for any f 2 C, for any distribution D on X , and for any 0 1 and 0 1, if L is given access to EX(f;D) and to inputs and , then with probability at least 1 , L outputs a hypothesis concept h satisfying error(h) D(f h) in time bounded by p(1= 1= n; size(f) The SQ model [Kea93] is a specialization of the PAC model in which the learner forms its hypothesis solely on the basis of estimates of probabilities. A statistical query over Xn is a mapping : Xn f0; 1g f0; 1g associated with a tolerance 0 1. In the SQ model the learner is given access to a statistics ....

....n; size(f) and L outputs a hypothesis h 2 C satisfying D(f h) It is clear that given access to the example oracle EX(f;D) it is easy to simulate the statistics oracle STAT (f; D) drawing a su ciently large set of labelled examples. This is formalized by the following result: Theorem 1. [Kea93] Let C be a class of concepts over X. Suppose that C is SQ learnable by algorithm L. Then C is PAC learnable, and furthermore: If L uses a nite query space Q and is a lower bound on the allowed approximation error for every query made by L, the number of calls of EX(f;D) is O(1= 2 log ....

[Article contains additional citation context not shown here]

M. Kearns. Ecient noise-tolerant learning from statistical queries. In Proceedings of the 25th ACM Symposium on the Theory of Computing, pages 392401. ACM Press, New York, NY, 1993.

....query case, sample points given to the oracle can be chosen, whereas for correlation attacks the sample points are simply randomly selected. The latter problem is actually a well known problem also in learning theory, called learning parity with noise , and it is commonly believed to be hard, see [11, 1]. Nevertheless, we are interested in nding as e cient correlation attacks as possible, and we will now derive an algorithm that is inspired by the results presented in the previous section. Let us rst brie y review our problem formulation. The recovery of the initial state of the target LFSR ....

M. Kearns, Ecient noise-tolerant learning from statistical queries, Proceedings of the Twenty-Fifth Annual ACM Symposium on Theory of Computing, San Diego, California, 16-18 May 1993, pp. 392401.

....errors that could be corrected by seeing su#cient number of instances. Recall that D may be a multiset and one example may appear in D more than once. Then by seeing enough number of instances of D for the same example, we may be able to fix it even if some instance is labeled wrongly. See, e.g. [27] for the formal argument. For solving this problem of AdaBoost, Freund [21] introduced a new weighting scheme and proposed a new boosting algorithm BrownBoost which is also robust to the statistical errors of the above type. Unfortunately, however, even these algorithms may not be robust ....

M. Kearns, E#cient noise-tolerant learning from statistical queries, in Proc. the 25th Annual ACM Sympos. on Theory of Comput. (STOC'93), 392--401, 1993.

....weighting system of AdaBoost. We first prove that one version of MadaBoost is in fact a boosting algorithm. Second, we show how our algorithm can be used and analyzed its performance in detail. Finally, we show that our new boosting algorithm can be casted in the statistical query learning model [Kea93] and thus, it is robust to random classification noise [AL88] This is a revised version of TR C133. 1 Introduction In the last decade, boosting techniques have been received a great deal of attention from the machine learning and computational learning communities. In this paper, we further ....

....was first proposed in [Wat99] with only a partial proof for its justification. In this paper, we describe the modification in detail, provide a much improved analysis of its correctness and performance, and prove that our new boosting algorithm can be casted in the statistical query learning model [Kea93] and thus, it is robust to random classification noise [AL88] as well as to some other kinds of less benign noise [Dec93] While the above problem (2) is an obvious weakness, it may not seem so important that even if the boosting algorithm does not work for the filtering framework , where ....

[Article contains additional citation context not shown here]

M. Kearns, E#cient noise-tolerant learning from statistical queries, In Proc. of the Twenty-Fifth Annual ACM Sympos. on Theory of Comput., 392--401, 1993.

....et al. 1989 ] The assumptions made here are also clearer than those implicit in Quinlan and Cameron Jones s [1995] measure. Freund [1998] recently proposed a form of processoriented evaluation that is closer to the PAC learning framework. It is an extension of the statistical query model [ Kearns, 1993 ] that attempts to obtain tighter bounds on generalization error by considering the tree of queries that the learner could make. While the general algorithm to obtain these bounds has exponential computational cost in the number of queries made, Freund proposes a specialized version for algorithms ....

M. Kearns. E#cient noise-tolerant learning from statistical queries. In Proceedings of the Twenty-Fifth ACM Symposium on the Theory of Computing, pages 392--401, New York, NY, 1993. ACM Press.

....perceptron networks. The algorithms work by estimating various statistical quantities that yield enough information to infer, with high probability, the target concept. Because of their statistical nature, these algorithms are robust against a large amount of random classification noise (Kearns, 1993). There are a number of previous algorithms for learning read once boolean formulas that exploit the idea of using various statistical estimates to infer the target function (Goldman et al. 1990; Pagallo Haussler, 1989; Schapire, 1991) While the high level structures of these statistical ....

....weights on the uniform distribution. These algorithms work by estimating statistical quantities that yield enough information to infer, with high probability, the target network. Because of their statistical nature, these algorithms are robust against a large amount of random classification noise (Kearns, 1993). The hardness results of (Blum Rivest, 1988; Judd, 1988) suggest that one can not avoid the training di#culties simply by considering only very simple neural networks. The results of this paper suggest that the combination of simple networks and reasonable distributions can overcome the ....

Kearns M. (1993). E#cient Noise-Tolerant Learning from Statistical Queries. Proceedings of the Twenty Fifth Annual ACM Symposium on the Theory of Computation, p. 392.

....arithmetic. In section 4 where we discuss in more detail the case where inputs come from the discrete boolean domain, we explain why this open problem is still likely to be hard. In this paper we show how to convert our algorithm into a statistical query (SQ) algorithm (as introduced by Kearns [28]) which implies that it can be made noise tolerant. Over the boolean domain f0; 1g d a more general result of this kind already exists, namely that learnability in the k RFA implies SQ learnability and hence learnability in the presence of random classi cation noise, for k logarithmic in d ....

...., 1 and d, but the runtime is exponential in d. We start by describing the algorithm, then give results to justify the steps. The algorithm is initially presented in the standard PAC setting. In section 3. 3 we show how to express it as a statistical query algorithm, as introduced by Kearns [28], who showed that such algorithms are noise tolerant. First we need the following de nition. De nition 13 The quadratic loss [29] of an example (x; l) with respect to a classi er C) where x is the input and l is a binary valued label, is the quantity (l P r(label = 1 j x; C) 2 , i.e. the ....

[Article contains additional citation context not shown here]

M.J. Kearns (1993). EÆcient Noise-Tolerant Learning From Statistical Queries, Procs. of the 25th Annual Symposium on the Theory of Computing, 392-401.

....forcing the weak learner to agree with them and thus, obtain erroneous hypothesis that degradate the generalization error of the combined hypothesis. From the theoretical side, Aslam and Decatur [2] showed that previous boosting algorithms belong to the statistical query learning model of Kearns [11], the model that is accepted as the most appropriate for studying whether a learning algorithm is robust to several kinds of noise, including the most commonly studied random classi cation noise [1] However, to the authors knowledge, no theoretical results about whether AdaBoost is robust to any ....

....We are planning to study experimentally this point in future work. For the time being, we will show here, that, from the theoretical point of view, MadaBoost is in fact resistant to certain kinds of noise by showing that it belongs to the statistical query model of learning introduced by Kearns [11]. Whether AdaBoost or any of its variants fall into this model or not is not known. Notice that one of the key point for showing this result is the fact that MadaBoost works in the ltering framework and thus, we can in fact estimate probabilities from the modi ed distributions. Before showing the ....

[Article contains additional citation context not shown here]

Michael Kearns. EÆcient noise-tolerant learning from statistical queries. In Proceedings of the Twenty-Fifth Annual ACM Symposium on Theory of Computing, 1993.

....the NLP context. 3 Robust Learning This section de nes a learning algorithm and a class of hypotheses with some generalization properties, that we later show to capture many probabilistic learning methods used in NLP. The learning algorithm discussed here is a Statistical Queries(SQ) algorithm [ Kearns, 1993 ] An SQ algorithm can be viewed as a learning algorithm that interacts with its environment in a restricted way. Rather than viewing examples, the algorithm only requests the values of various statistics on the distribution of the labeled examples to construct its hypothesis. e.g. What is the ....

.... labeled examples (x; l) according to D and evaluate P r S [ x; l) jf(x; l) x; l) 1jg jSj : Cherno bounds guarantee that the number of examples required to achieve tolerance with probability at least 1 is polynomial in 1= and log 1= The SQ model of learning was introduced by Kearns [ 1993 ] and studied in [ Decatur, 1993; Aslam and Decatur, 1995 ] It was viewed as a tool for demonstrating that a pac learning algorithm is noise tolerant. In particular, it was shown that learning with an SQ algorithm allows the learner to tolerate examples with noisy labels labels which, with ....

M. Kearns. EÆcient noise-tolerant learning from statistical queries. In Proceedings of the Twenty-Fifth Annual ACM Symposium on Theory of Computing, pages 392-401, 1993.

....could be caused by noisy communication, human errors, measuring equipment and many other causes. In some cases, problems which are e ciently learnable without noise, become hard to learn when noise is introduced [3] Partial answers to the noisy learning problem are known for speci c classes, [11]) but in general, no simple parameters are known which distinguish between classes that are learnable in the presence of noise and those which become hard to learn. The goal of this work is to introduce such parameters. We use the Membership Query model to show that if the VC dimension of the ....

....While the Vapnik Chervonenecis dimension ( 19] 4] completely characterizes PAC learnability of classes of f0; 1g functions, the case of learning in a noisy environment is unresolved. Nevertheless, in a variety of cases one can show that even in the presence of noise learning is possible. Kearns [11] presented the model of Statistical Queries (SQ) and proved that if a class is learnable with SQs then it is also learnable in the presence of random labeling noise and presented a variety of problems in which this approach may be applied. Since Kearns pioneering paper, much work was done on the ....

M. J. Kearns. EÆcient noise-tolerant learning from statistical queries. In the proceedings of the Twenty-Fifth Annual ACM Symposium on Theory of Computation, pages 392-401, 1993.

....Holte et al. 1989 ] The assumptions made here are also clearer than those implicit in Quinlan and Cameron Jones s [1995] measure. Freund [1998] recently proposed a form of processoriented evaluation that is closer to the PAC learning framework. It is an extension of the statistical query model [ Kearns, 1993 ] that attempts to obtain tighter bounds on generalization error by considering the tree of queries that the learner could make. While the general algorithm to obtain these bounds has exponential computational cost in the number of queries made, Freund proposes a specialized version for algorithms ....

M. Kearns. EÆcient noise-tolerant learning from statistical queries. In Proceedings of the Twenty-Fifth ACM Symposium on the Theory of Computing, pages 392-401, New York, NY, 1993. ACM Press.

No context found.

Kearns, M.: E#cient noise-tolerant learning from statistical queries. Journal of the ACM 45 (1998) 983--1006

No context found.

M.J. Kearns. E#cient Noise-Tolerant Learning from Statistical Queries. Journal of the ACM, 45(6):983-1006, 1998.

No context found.

M.J. Kearns. E#cient Noise-Tolerant Learning from Statistical Queries. Journal of the ACM, 45(6):983-1006, 1998.

No context found.

M. Kearns, Ecient noise-tolerant learning from statistical queries, Proceedings of the Twenty-Fifth Annual ACM Symposium on Theory of Computing, San Diego, California, 16-18 May 1993, pp. 392401.

No context found.

M.J. Kearns. E#cient Noise-Tolerant Learning from Statistical Queries. Journal of the ACM, 45(6):983-1006, 1998.

No context found.

M.J. Kearns. E#cient Noise-Tolerant Learning from Statistical Queries. Journal of the ACM, 45(6):983-1006, 1998.

No context found.

Michael Kearns. E#cient Noise-Tolerant Learning from Statistical Queries. In Proceedings of the Forth Annual Workshop on COLT, pp. 392--401, 1993.

No context found.

M. Kearns. E#cient noise-tolerant learning from statistical queries. Journal of the ACM, 45(6):983--1006, 1998.

No context found.

M. Kearns. E#cient noise-tolerant learning from statistical queries. Journal of the ACM, 45(6):983--1006, 1998.

No context found.

M. Kearns. E#cient noise-tolerant learning from statistical queries. In Proceedings of the Twenty-Fifth ACM Symposium on the Theory of Computing, pages 392--401, New York, NY, 1993. ACM Press.

No context found.

M. Kearns. Ecient noise-tolerant learning from statistical queries. In Proc. 25th ACM Symposium on the Theory of Computing, pages 392  401, 1993.

No context found.

Michael Kearns. Ecient Noise-Tolerant Learning From Statistical Queries, Proceedings of the 25th ACM Symposium on the Theory of Computing, pp. 392-401, 1993, ACM Press. http://www.research.att.com/mkearns/papers/sq-noise.ps.Z

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC