Ming Li and P. M. B. Vitanyi. A theory of learning simple concepts under simple distributions and average case complexity for the universal distribution. In Proc. of the 30th IEEE Annual Symp. on Foundation of Computer Science, pages 34--39, 1989.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....results, except they concern distributional problems hproblem, input distributioni pairs. These results relate the complexities of classes of distributional problems. Generally, few relations to worst case complexity are known (see, however, 4] Ben David et al. 4] and Li and Vitanyi [13] show the existence of distributions under which the average case complexity of any program is within a constant (exponential in the size of the program) of the worst case complexity. Generating random instances from such distributions is difficult it requires diagonalizing against all ....

Ming Li and P. M. B. Vitanyi. A theory of learning simple concepts under simple distributions and average case complexity for the universal distribution. In Proc. of the 30th IEEE Annual Symp. on Foundation of Computer Science, pages 34--39, 1989.

....learning algorithm for H learns with respect to and has polynomial sample complexity. For further discussion of distribution dependent learning, we refer the reader to the papers of Benedek and Itai [32] Ben David, Benedek and Mansour [27] Bertoni et al. 33] Kharitonov [66] Li and Vitanyi [68], Linial, Mansour and Nisan [69] Graph Dimension and Multiple Output Nets 42 11 Graph Dimension and Multiple Output Nets The basic PAC model concerns learning f0; 1g valued functions only; that is, it is concerned only with classification problems. A significant and important extension of the ....

M. Li and P. M. B. Vitanyi. A theory of learning simple concepts under simple distributions and average case complexity for the universal distribution. In Proc. 30th Annu. IEEE Sympos. Found. Comput. Sci., pages 34--39. IEEE Computer Society Press, Los Alamitos, CA, 1989. REFERENCES 66

....may be computationally feasible in situations where standard pac learning is NP hard. For further discussion of distribution dependent learning, we refer the reader to the papers of Benedek and Itai (1992) Ben David, Benedek and Mansour (1989) Bertoni et al. 1992) Kharitonov (1993) Li and Vitanyi (1989), Linial, Mansour and Nisan (1989) In the standard pac framework, the learning algorithm receives labelled examples and forms a hypothesis only on the basis of these. The learning algorithm has no control over the sequence of training examples. Clearly, it might be possible to make learning ....

Li and Vitanyi (1989): M. Li and P. Vitanyi, A theory of learning simple concepts under simple distributions and average case complexity for the universal distribution. In Proceedings of the Thirtieth IEEE Symposium on Foundations of Computer Science, IEEE Computer Society Press, Washington DC.

....constructor. 1 Introduction The average case time complexity of specific algorithms has for a number of years been an active area of research, often showing significant improvement over the worst case complexity when specific distributions of the inputs were assumed. Recently, Li and Vit anyi [4] studied the Solomonoff Levin measure m and found that when the inputs to any algorithm are distributed according to this measure, it holds that the algorithm s average case complexity is of the same order of magnitude as its worst case complexity. More precisely, for some c 0 X x2 Sigma n ....

....contract No. 3075 (project ALCOM) ffl The Solomonoff Levin measure assigns large amounts of probability to strings with lots of pattern, and small amounts to random strings. Therefore, the result seems to imply that worst case strings or strings close to worst case will be easily describable. In [4], the example of quicksort is given, where the worst case strings are the sorted or almost sorted ones. These have short descriptions, and hence high Solomonoff Levin measure. That worst case strings in general are easily describable seems counterintuitive. ffl Li and Vit anyi argue in [5] that ....

[Article contains additional citation context not shown here]

M. Li, P.M.B. Vit'anyi, A theory of learning simple concepts under simple distributions and average case complexity for the universal distribution, in Proc. 30th Annual Symposium on Foundations of Computer Science, Research Triangle Park, NC, October 1989, pp. 34-39.

.... in (Watanabe, 1992) Easily computable approximations of the MDL principle were formulated by Wallace and Boulton (1968) and Rissanen (1978, 1983, 1986) Such approximations build the basis of most if not all current machine learning applications, e.g. Quinlan and Rivest, 1989; Gao and Li, 1989; Milosavljevi c and Jurka, 1993; Pednault, 1989) Barzdin, referred to in (Zvonkin and Levin, 1970) see also Barzdin, 1988) related Kolmogorov complexity to a variant of Godel s incompleteness theorem, a subject which became a central theme of Chaitin s research (Chaitin, 1987) Meanwhile, the ....

....sequence compression. Becker, 1991) provides numerous additional references. There are also numerous heuristic constructive methods, where network size grows in case of underfitting the training data. MDL approaches in other areas of machine learning include (Quinlan and Rivest, 1989; Gao and Li, 1989; Milosavljevi c and Jurka, 1993; Pednault, 1989) Among the implemented methods, neither the neural net approaches nor the other ones are general in the sense of Solomonoff, Kolmogorov, and Levin. All the previous implementations use measures for simplicity that lack the universality and ....

[Article contains additional citation context not shown here]

Li, M. and Vit'anyi, P. M. B. (1989). A theory of learning simple concepts under simple distributions and average case complexity for the universal distribution. In Proc. 30th American IEEE Symposium on Foundations of Computer Science, pages 34--39.

....learning algorithm for a very simple class of feedforward linear threshold networks. In a sense, since such negative results depend on the means by which output hypotheses are represented, they are perhaps not strong enough. Modulo certain standard cryptographic hardness assumptions, Kearns and Valiant (1989) proved that certain concept spaces are not efficiently learnable by any reasonable hypothesis space. We are being rather vague here and refer the reader to their paper for details. In the standard pac learning framework, the learning algorithm receives labelled examples at random and has no ....

....learning algorithm for H learns with respect to and has polynomial sample complexity. For further discussion of distribution dependent learning, we refer the reader to the papers of Benedek and Itai (1992) Ben David, Benedek and Mansour (1989) Bertoni et al. 1992) Kharitonov (1993) Li and Vitanyi (1989), Linial, Mansour and Nisan (1989) 7. GENERALISING THE VC DIMENSION FOR FUNCTION SPACES The basic pac model concerns learning f0; 1g valued functions only; that is, it is concerned only with classification problems. A significant and important extension of the basic pac model is to the learning ....

Li and Vitanyi (1989): M. Li and P. Vitanyi, A theory of learning simple concepts under simple distributions and average case complexity for the universal distribution.

.... allowing exponential time to the adversary since a theorem of [1] shows that for every NP problem there are distributions, computable in exponential time, for which the problems average case and worst case complexity are similar (similar results for general computable problems are exposed in [26]) Levin s original definition limits the power of the adversary by requiring the distribution of hS; i (x) def = Pr fy S xg ( S a lexicographical order on S) to be polynomial time computable. More general is to allow hS; i to be polynomial time sampleable which means that there ....

M. Li and P. M.B. Vitanyi. A theory of learning simple concepts under simple distributions and average case complexity for the universal distribution. In Proc. 30th Symposium on Foundations of Computer Science, pages 34--39, 1989. (Extended Abstract).

.... contextsensitive grammars become learnable with positive data if the learner knows a bound on the number of rules in the grammar ( 24] Another approach consists in putting structural, statistical or complexity constraints on the examples proposed to the learner, making his her induction easier ([15, 21]) This solution formalizes the help provided by a professor ( 6] A particular family of research, more concerned with the cognitive relevance of its models, considers that learning a natural language is very different from learning a formal language, because in natural situations, examples are ....

M. Li, P. Vitanyi, "A theory of learning simple concepts under simple distributions", SIAM J. Computing, 20(5), p915-935, 1991.

....in (Watanabe, 1992) for more. Easily computable approximations of the MDL principle were formulated by Wallace and Boulton (1968) and Rissanen (1978, 1983, 1986) Such approximations build the basis of most if not all current machine learning applications, e.g. Quinlan and Rivest, 1989; Gao and Li, 1989; Milosavljevi c and Jurka, 1993; Pednault, 1989) Barzdin, referred to in (Zvonkin and Levin, 1970) related Kolmogorov complexity to a variant of Godel s incompleteness theorem, a subject which became a central theme of Chaitin s research (Chaitin, 1987) Meanwhile, the theory of Kolmogorov ....

....sequence compression. See (Becker, 1991) for numerous additional references. There are also numerous heuristic constructive methods, where network size grows in case of underfitting the training data. MDL approaches in other areas of machine learning include (Quinlan and Rivest, 1989; Gao and Li, 1989; Milosavljevi c and Jurka, 1993; Pednault, 1989) Among the implemented methods, neither the neural net approaches nor the other ones are general in the sense of Solomonoff, Kolmogorov, and Levin. All the previous implementations use measures for simplicity that lack the universality and ....

[Article contains additional citation context not shown here]

Li, M. and Vit'anyi, P. M. B. (1989). A theory of learning simple concepts under simple distributions and average case complexity for the universal distribution. In Proc. 30th American IEEE Symposium on Foundations of Computer Science, pages 34--39.

....in Section 4 below, to the terminology of DNF formulae; however, the resulted subclasses of DNF seem to be somewhat artificial. Example: Consider the class of all DNF formulae in which each term contains at least n Gamma k literals (for some constant k) This class was previously considered in [26]. In spite of its similarity to the class of k DNF formulae (where each term contains at most k literals) its learnability seems to be more difficult. This is because in this case there are exponentially many possible monomials to choose from (as opposed to the case of k DNF where there are only ....

M. Li and P. M. B. Vitanyi. A theory of learning simple concepts under simple distributions and average case complexity for the universal distribution. In Proceedings of the Thirtieth Annual Symposium on Foundations of Computer Science, pages 34--39, Research Triangle Park, North Carolina, October 1989.

....distribution, they it is learnable under all computable distributions. Formally, Theorem 2 A concept class C is polynomially learnable under the universal distribution m(x) iff it is polynomially learnable under each computable distribution P , provided the sample is drawn according to m. See [12] for details. In the continuous case, we even have a stronger theorem without needing to sample according to the universal distributions. Theorem 3 A concept class C over a continuous sample space is learnable under M iff it is learnable under each computable measure. 5 Can we abandon pumping ....

M. Li and P.M.B. Vit'anyi. A theory of learning simple concepts under simple distributions. SIAM J. Computing, 20(5):915--935, 1991.

....there are may even be helpful teachers. Great efforts have been made to improve the pac model to be more close to practical situations. New models are proposed to allow the learner to ask various kinds of questions, to have good teachers [1] to learn under a restricted class of distributions [11]. 4 Learning With MDL The previous models were mainly about formal criteria for learning such as learning in the limit and pac learning. The notion of learning by enumeration is an algorithm for learning, albeit a very unpractical one. Similarly, the MDL principle is an algorithmic paradigm ....

M. Li and P. Vit'anyi, A theory of learning simple concepts under simple distributions, SIAM J. Computing, 20:5(1991), 915-935.

....learnable Maybe one can just learn a concept under some distributions, like the computable ones. And maybe it is too much to ask to be able to learn all finite automata fast (humans cannot either) but surely we ought to be able to learn a sufficiently simple finite automaton fast (as humans can) [6]. Certain concepts which are not know to be polynomially pac learnable in the distribution free model, can be polynomially pac learned under the uniform distribution by a specialized learning algorithm. However, learning under one distribution may be too restrictive to be useful. It may be useful ....

....polynomial pac learnability under various classes of distributions. One would like this class to be wide enough to be interesting, yet restrictive enough to make more concept classes polynomially learnable. It is possible to develop a theory of simple pac learning in the following sense [6]. Discrete sample set. Let Q be a distribution on a discrete sample set. A concept class is polynomially pac learnable under all distributions which are multiplicatively dominated by Q provided you draw your sample according to Q in the learning phase iff the concept class is polynomially ....

M. Li and P.M.B. Vit'anyi. A theory of learning simple concepts under simple distributions. SIAM J. Computing, 20(5):915--935, 1991.

No context found.

M. Li and P.M.B. Vitanyi. A theory of learning simple concepts under simple distributions and average case complexity for the universal distribution. In Proceedings of the 30th IEEE Symposium on Foundations of Computer Science, pages 34--39, 1989.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC