Hans Ulrich Simon. General bounds on the number of examples needed for learning prob- abilistic concepts. In Proceedings of the Sixth Annual A CM Workshop on Computational Learning Theory, pages 402-411. ACM Press, 1993.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....constructing SQ algorithms which are nearly optimal with respect to these bounds. However, the robust PAC learning algorithms obtained by sim ulating even optimal SQ algorithms in the presence of noise are inefficient when compared to known lower bounds for PAC learning in the presence of noise [11, 20, 30]. In fact, the PAC learning algorithms obtained by simulating optimal SQ algorithms in the absence of noise are inefficient when compared to the tight bounds known for noise free PAC learning [7, 11] These shortcomings could be consequences of either inefficient simulations or a deficiency in the ....

....affected by the branching factor of the simulation. By reducing i i 1 the branching factor of the simulation from O( 1 2)b)2 ) to (W log 1) the asymptotic running time of our simulation is greatly improved. 1 With respect to 7, the running time of our simulation is O( 1 ) Simon [30] has shown a sample and time complexity lower bound of f( 1) for PAC learning in the presence of classification noise. We therefore note that the running time of our simulation is optimal with respect to the noise rate (modulo lower order logarithmic factors) For dynamic algorithms, the time ....

Hans Ulrich Simon. General bounds on the number of examples needed for learning prob- abilistic concepts. In Proceedings of the Sixth Annual A CM Workshop on Computational Learning Theory, pages 402-411. ACM Press, 1993.

....for constructing SQ algorithms which are nearly optimal with respect to these bounds. However, the robust PAC learning algorithms obtained by simulating even optimal SQ algorithms in the presence of noise are inefficient when compared to known lower bounds for PAC learning in the presence of noise [11, 20, 30]. In fact, the PAC learning algorithms obtained by simulating optimal SQ algorithms in the absence of noise are inefficient when compared to the tight bounds known for noise free PAC learning [7, 11] These shortcomings could be consequences of either inefficient simulations or a deficiency in the ....

....by the branching factor of the simulation. By reducing the branching factor of the simulation from Theta( 1 Gamma2j b ) 2 ) to Theta( the asymptotic running time of our simulation is greatly improved. With respect to j, the running time of our simulation is O( 2 ) Simon [30] has shown a sample and time complexity lower bound of Omega Gamma (1 Gamma2j) 2 ) for PAC learning in the presence of classification noise. We therefore note that the running time of our simulation is optimal with respect to the noise rate (modulo lower order logarithmic factors) For dynamic ....

Hans Ulrich Simon. General bounds on the number of examples needed for learning probabilistic concepts. In Proceedings of the Sixth Annual ACM Workshop on Computational Learning Theory. ACM Press, 1993.

....PAClearnable under any (D; R) noise in time polynomial in jT j, 1= 1 Gamma 2j) and 1= Delta, where j is the expectation of the noise rate in the label. Since 1= Delta is a lower bound for PAClearning with D distribution noise and 1= 1 Gamma 2j) is a lower bound for learning with label noise [9], our result is tight (up to polynomial factors) We now formally state the result: Theorem 7 Let C be a class of Boolean functions and suppose that C is learnable with respect to the uniform distribution by an LMN style algorithm using index set T ffl . Then for every ffl such that the set of ....

Hans Ulrich Simon. General Bounds on the Number of Examples Needed for Learning Probabilistic Concepts. In Proceedings of the 6th Annual ACM Workshop on Computational Learning Theory, pages 402-411, 1993.

....for constructing SQ algorithms which are nearly optimal with respect to these bounds. In spite of this, the robust PAC learning algorithms obtained by simulating SQ algorithms in the presence of noise are inefficient when compared to known lower bounds for PAC learning in the presence of noise [8, 13, 18]. In fact, the PAC learning algorithms obtained by simulating SQ algorithms in the absence of noise are inefficient when compared to the tight bounds known for noise free PAC learning [6, 8] These shortcomings could be consequences of either inefficient simulations or a deficiency in the model ....

....SQ algorithms may submit various queries based upon the estimates they receive for previous queries. We refer to these algorithms as dynamic SQ algorithms. 2 Note that multiple runs of a dynamic SQ algorithm may produce many more queries which need to be estimated. With respect to j, Simon [18] has shown a sample complexity, and therefore time complexity, lower bound of Omega Gamma 1 (1 Gamma2j) 2 ) for PAC learning in the presence of classification noise. We therefore note that by reducing the branching factor of the simulation from Theta( 1 (1 Gamma2j b ) 2 ) to Theta( 1 ....

Hans Ulrich Simon. General bounds on the number of examples needed for learning probabilistic concepts. In Proceedings of the Sixth Annual ACM Workshop on Computational Learning Theory, pages 402--411. ACM Press, 1993.

.... 2 (1 Gamma 2j b ) 2 1 0 2 ffl 2 (1 Gamma 2j b ) 2 log 1 ffi : One can also show that the dependence on ffl and j b of the space complexity of learning F is O( 1 ffl 2 (1 Gamma2j b ) 2 ) while the dependence on ffl and j b of the hypothesis size is O(log 1 ffl ) Simon [17] has shown that the sample complexity of PAC learning with classification noise is Omega Gamma 1 ffl(1 Gamma2j) 2 ) Therefore our sample complexity bound is roughly optimal with respect to j b . Note that the hypothesis size is independent of the noise rate. Furthermore, with a polynomial ....

Hans Ulrich Simon. General bounds on the number of examples needed for learning probabilistic concepts. In Proceedings of the Sixth Annual ACM Workshop on Computational Learning Theory. ACM Press, 1993.

....take an algorithm which tolerates classification noise, and use this technique to create one which tolerates CAM error. The amount of CAM error which can be tolerated using these techniques may be upper bounded by using lower bounds on the sample complexity of classification noise learning. Simon [Sim93] has shown that any algorithm for function class F which tolerates classification noise must have sample complexity at least m = Omega Gamma VCDim(F) 1=2 Gammaj) 2 ) 4 In fact, most known classification noise algorithms have sample complexity at least m = Omega Gamma VCDim(F) 2 ....

Hans Ulrich Simon. General bounds on the number of examples needed for learning probabilistic concepts. In Proceedings of the Sixth Annual ACM Workshop on Computational Learning Theory, pages 402--411. ACM Press, 1993.

....showed that a specific learning algorithm, one which simply chooses the function in the target class F with the fewest disagreements on the sample of data, requires Omega Gamma log(jF j=ffi) 1 Gamma2j) Delta examples. Note that this result is only applicable to finite target classes. Simon [9] showed that any algorithm for learning in the presence of classification noise requires Omega Gamma VC(F) 1 Gamma2j) 2 Delta examples where VC(F) is the Vapnik Chervonenkis dimension of F . 1 One could also consider a general lower bound on the sample complexity of noise free learning ....

....on a sample of size m, the algorithm fails with probability at most ffi when the target is f 0 , then the algorithm must fail with probability more than ffi when the target is f 1 . 2 Note that we have not attempted to optimize the constants in this lower bound. 3. 1 The Combined Lower Bound Simon [9] proves the following lower bound: Theorem 2 PAC learning a function class F in the presence of classification noise requires a sample of size Omega Gamma VC(F) 1 Gamma2j) 2 Delta . Combining Theorems 1 and 2 we obtain the following lower bound on the number of examples required for ....

Hans Ulrich Simon. General bounds on the number of examples needed for learning probabilistic concepts. In Proceedings of the Sixth Annual ACM Workshop on Computational Learning Theory, pages 402--411. ACM Press, 1993.

....3.4) by showing that at least order of = Delta 2 d= Delta examples are needed to PAC learn, with accuracy and tolerating a malicious noise rate j = 1 ) Gamma Delta, every class of f0; 1g valued functions of VC dimension d. Our proof combines, in an original way, techniques from [2, 4, 10] and uses some new estimates of the tails of the binomial distribution that may be of independent interest. We then prove that this lower bound cannot be improved in general. Namely, we show (Theorems 3.18 and 3.10) that there is an algorithm RMD (for Randomized Minimum Disagreement) that, for ....

Hans Ulrich Simon. General bounds on the number of examples needed for learning probabilistic concepts. In Proceedings of the 6th Annual Workshop on Computational Learning Theory, pages 402--412. ACM Press, 1993. To appear in Journal of Computer and System Sciences.

....3.4) by showing that at least order of = Delta 2 d= Delta examples are needed to PAC learn, with accuracy and tolerating a malicious noise rate j = 1 ) Gamma Delta, every class of f0; 1g valued functions of VC dimension d. Our proof combines, in an original way, techniques from [5, 8, 15] and uses some new estimates of the tails of the binomial distribution that may be of independent interest. We then prove that this lower bound cannot be improved in general. Namely, we show (Theorem 3.10 and Corollary 3.15) that there is an algorithm rmd (for Randomized Minimum Disagreement) ....

....Forschungsgemeinschaft research grant We 1066 6 2. The authors take the responsibility for the contents. Thanks to an anonymous reviewer for pointing out [12] 5 See e.g. 11] for a survey on this noise model and for the upper bound on the sample size in the case of finite concept classes. See [15] for the lower bound on the sample size, and [16] for a generalization of Laird s upper bound to arbitrary concept classes. ....

Hans Ulrich Simon. General bounds on the number of examples needed for learning probabilistic concepts. Journal of Computer and System Sciences, 52(2):239--255, 1996.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC