A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. Journal of Computer and System Sciences, 50:32--40, 1995.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....algorithms whose query complexity is polynomial in the syntactic difference (or revision distance) between the initial theory and the target theory, but only polylogarithmic in the total number of possible variables. Thus, this work has some similarities to the work on attribute efficient learning [3, 4]. A particular measure of revision distance is determined by fixing a specific set of elementary operations, which we will call revision operators. Following the spirit of much work in machine learning on theory revision, we consider two sets of revision operators, the deletions only revision ....

A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. J. of Comput. Syst. Sci., 50(1):32--40, 1995.

....how well it works in practice. In general, the additional work described in section 6.6 on monitoring is a good direction for future work. The most interesting direction for work on CARD is automatic derivation of dependencies. The idea here is to use either machine learning [AL88, Kea93, BHL91, KL88, Lit89] or association rule mining [AIS93, AS94] techniques to automatically determine dependencies. This approach requires having some monitored values that indicate if a system is up or down. Then, if we can show that any time component 1 is down, component 2 is also down, but not the ....

A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. In Proceedings of the Fourth Annual Workshop on Computational Learning Theory, pages 157--166, Santa Cruz, California, August 1991. Morgan Kaufmann.

....out by the brain (Marr [47] Extracting that which is essential is likewise a difficult algorithmic problem arising in various circumstances the clique problem, traveling salesman and many more. In the context of computational learning it was addressed explicitly by Blum [16] Blum et al. [15], Littlestone [43] Ben David and Dichterman [10, 11] Birkendorf et al. 14] Consider a manager faced with the task of hiring experts from a pool of N candidates. We assume that he can find out the utility of hiring particular sets of experts by querying an oracle x : 2 . Finding an ....

A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. In Proceedings of the Fourth Annual Workshop on Computational Learning Theory, pages 157--166. Morgan Kaufmann, August 1991.

....particularly useful when the target concept depends on few variables but n; the total number of variables, is large. Results of Haussler [16] and Littlestone [22] yield attribute efficient learning algorithms for k CNF and k DNF formulae; more recent results on attribute efficiency can be found in [4, 7, 28]. Blum [2] and Valiant [30] have each posed the question of whether there exists a polynomial time attributeefficient learning algorithm for the concept class of 1 decision lists of length k: Such an algorithm would run in time poly(n) this is unavoidable since each example is of length n) but ....

....y; z 2 f0; 1g n which have y j = z j for all j 6= i; y i 6= z i ; and c(y) 6= c(z) Let C be a class of boolean functions on x 1 ; x n each of which depends on at most r variables and each of which has a description of length at most s under some reasonable encoding scheme. Following [4], we say that a learning algorithm L for C in the mistake bound model is attribute efficient if the mistake bound of L on any concept c 2 C is poly(r; s; log n) In this section we provide strong evidence that there are concept classes learnable in polynomial time for which attribute efficient ....

A. Blum, L. Hellerstein, and N. Littlestone, Learning in the presence of finitely or infinitely many irrelevant attributes, J. Comput. System Sci. 50 (1995), 32--40.

....classifier to estimate the probability that a document is liked. Both of these approaches have a shortcoming in that they require prespecifying the number of terms used in the profile. In this research, we take an alternate approach by using the Winnow algorithm (Littlestone Warmuth, 1994; Blum, Hellerstein Littlestone, 1995). Winnow is designed to identify relevant features when there are many possible attributes. Prior experimental research has demonstrated that Winnow works well on text classification (Lewis, Schapire, Callan, Papka, 1996; Blum, 1997) in which each word x i (or pair of adjacent words) is treated ....

Blum, A., Hellerstein, L. & Littlestone, N. (1995). Learning in the Presence of Finitely or Infinitely Many Irrelevant Attributes. Journal of Computer and System Sciences 50(1): 3240.

....new relevant variable. Definition 6. The virtual target f V on a set V of variables about the target concept f is a Boolean function f V with rel(f V ) V such that fV (AV ) f(AV ) holds for any instance A. In the mistake bound model with membership queries, Blum, Hellerstein and Littlestone [4] measures ProbA ff(A) h V (A)g for a temporal hypothesis h to find a new relevant variable that is not yet implemented in h. Here we measure ProbA ff(A) f V (A)g, the distance between f and f V . If it is large, then our algorithm finds a witness for relevance (A; A V ) with high probability. ....

A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. Journal of Computer and System Sciences, 50(1):32--40, 1995.

.... or vote, and the class that gets the highest overall vote is re 2 Strictly speaking, we are making use of the basic Winnow II strategy [Littlestone, 1988] Since, over time, the number of specialists can increase unboundedly, we are using the infinite attribute model by Blum [Blum, 1992; Blum et al. 1991] turned as the final class. The goal of learning is to find a suitable weighting scheme for each expert. In this work each specific expert corresponds to some property of a message. For a new message it consults the last five times messages with this property were previously seen (or fewer if ....

Avrim Blum, L. Hellerstein, and Nicholas Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. In Proceedings of the Fourth Annual Workshop on Computational Learning Theory, pages 157--166, Santa Cruz, California, 1991. Morgan Kaufman.

....the general case and results for restricted cases can be derived from ones presented here. n is much larger than the number of positive attributes any one example has, and an example is presented as a list of its positive attributes. Concept learning in this model is studied in (Blum, 1992; Blum, Hellerstein, and Littlestone, 1991). For the task of learning a world representation in order to reason about it later, it seems that the agnostic approach, the existential interpretation (2) ought to be taken. Several other works (Ben David and Dichterman, 1993; Greiner, Grove, and Kogan, 1996; Schuurmans and Greiner, 1994) ....

Blum, A., L. Hellerstein, and N. Littlestone. 1991. Learning in the presence of finitely or infinitely many irrelevant attribute. In Proceedings of the Annual ACM Workshop on Computational Learning Theory, pages 155--166.

....of the message, a tokenization of the fields: from, to, a union of to and cc, and the subject. 1 Strictly speaking, we are making use of the basic Winnow II strategy [8] Since, over time, the number of specialists can increase unboundedly, we are using the infinite attribute model by Blum [1, 3]. Table 1: Dataset properties of messages forwarded in User Size full set first 2 3 last 1 3 AD 1730 13.70 11.43 18.22 HH 5651 17.02 16.67 17.72 SM 1526 10.41 10.69 9.86 2.4 Evaluation The evaluation used in this paper is the break even point of a method. This method attempts ....

A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. In Proceedings of the Fourth Annual Workshop on Computational Learning Theory, pages 157--166, Santa Cruz, California, 1991. Morgan Kaufman.

....The coefficients defining the linearly separable concept, as well as the number of nonzero coefficients, can be bounded by a polynomial of the number of rules. Hence, Littlestone s Winnow2 algorithm [8] can be used to learn k level rule sets. By applying the transformation given by Blum et al. [2], this approach can also be made to work with infinite basic concept classes. The resulting algorithm learns the same concepts as our algorithm and has the advantage of being an on line mistake bounded algorithm. This method was pointed to the authors by Avrim Blum. The advantages of our ....

A. Blum, L. Hellerstein, and N. Littlestone, Learning in the presence of finitely or infinitely many irrelevant attributes, Proc. 4th Annual Workshop on Computational Learning Theory, Morgan Kaufmann, San Mateo, 1991, pp. 157--166.

....sky at that moment is irrelevant. The approach (3) is useful when the total number of attributes n is much larger than the number of positive attributes any one example has, and an example is presented as a list of its positive attributes. Concept learning in this model is studied in (Blum, 1992; Blum, Hellerstein, and Littlestone, 1991). For the task of learning a world representation in order to reason about it later, it seems that the agnostic approach, the existential interpretation (2) ought to be taken. A motivating scenario is that of an agent who is wandering around in the world, but can perceive at any instance only a ....

Blum, A., L. Hellerstein, and N. Littlestone. 1991. Learning in the presence of finitely or infinitely many irrelevant attribute. In Proceedings of COLT '91.

....queries, nor is it known whether decision trees are properly PAC learnable with or without membership queries. In this paper, we focus on concepts that depend on very few variables. Since Littlestone s seminal work [17] on this topic, such concept classes have been well studied in learning theory [6, 7, 10]. Recently, Damaschke [8, 9] studied exact learning of Boolean functions when irrelevant attributes abound, primarily in the model of learning with membership queries alone. He was able to show that a set X of nonadaptive membership queries can be constructed in time polynomial in n, the number of ....

A. Blum, L. Hellerstein, N. Littlestone. Learning in the Presence of Finitely or Infinitely Many Attributes. Journal of Computer and System Science, pages 50:32--40, 1995.

....ill defined. Such vague boundaries are not captured well by traditional rule based category descriptors. Notice also that the notion of probabilistic concept, or p concept (Kearns Schapire, 1994) does not capture this phenomenon, although fuzzy boundary models may (Angluin Slonim 1994; Blum et al. 1995; Frazier et al. 1996) The issue of typicality of an example relative to a category has been well investigated. Rosch Mervis (1975) demonstrated that the more features an example had that were judged as relevant to category membership, the higher typicality rating the example received. These ....

.... conjunctive concept) suggests interesting extensions: Is there an algorithm that learns in time that depends only logarithmically on the number of irrelevant features What if the feature space is infinite Notice that this model is quite different than the infinite attribute space model (Blum et al. 1995) because in the latter case, a positive example must contain features relevant for classification. The example above addresses only how some features present in an example can bring attention to other, missing ones, via deduction using a theory. What about a richer model that can address how ....

Blum, A., Hellerstein, L., Littlestone, N. (1995). Learning in the Presence of Finitely or Infinitely Many Irrelevant Attributes. JCSS, 50, 32-40.

....sufficient to establish diseaseX, etc. John et al. JKP94] would consider t 1 to be weakly irrelevant ; by contrast, an attribute is strongly irrelevant if its value never plays a role in the classification, under any circumstance (i.e. independent of the values of any other attributes) cf. [Lit88, Blu92, BHL91]. Of course, our situation differs from those models of learning, as our environment explicitly identifies which attributes are weakly irrelevant. Two final comments to help place our model within the framework of existing computational learning results: First, in our model, certain attribute ....

A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. In Proc. 4th Annu. Workshop on Comput. Learning Theory, pages 157--166. Morgan Kaufmann, San Mateo, CA, 1991.

....assumptions about the domains in which the machine learning system is working. If the learning system is allowed to make queries about the outputs corresponding to arbitrary input vectors, then the determination of smallest possible feature sets for discrete data becomes completely tractable [3]. If it is known that the function being learned is of a certain class (e.g. K CNF or K DNF) then the problem once again becomes tractable [2] ....

Blum, A.; Hellerstein, L.; and Littlestone, N. 1991. "Learning in the Presence of Finitely or Infinitely Many Irrelevant Attributes," Proceedings of the Fourth Annual Workshop on Computational Learning Theory. Santa Cruz, CA: Morgan Kaufman.

....etc. John et al. JKP94] would therefore consider x 1 to be weakly irrelevant ; by contrast, they say an attribute is strongly irrelevant if its value never plays a role in the classification, under any circumstance (i.e. independent of the values of any other attributes) cf. Lit88, Blu92, BHL91] Our situation differs from those models of learning as our environment explicitly identifies which attributes are weakly irrelevant in each instance. Our final comments help to place our model within the framework of existing computational learning results: First, in our model, certain ....

A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. In Proc. 4th Annu. Workshop on Comput. Learning Theory, pages 157--166. Morgan Kaufmann, San Mateo, CA, 1991.

....attribute efficiently in polynomial time. We show that this does not hold in the randomized membership query model. In the mistakebound model, we consider the problem of learning attribute efficiently using hypotheses that are formulas of small depth. Our results extend the work of Blum et al. [4] and Bshouty et al. 7] 1 Introduction Consider the problem of learning an unknown Boolean function on n variables where n is large. Suppose that the output of this function is completely determined by the values of a fixed set of r of the n variables, where r is small. Thus our real task is ....

....which has only a sublinear dependence on the number of irrelevant attributes in the target function. Littlestone developed a polynomial time, log n attribute efficient algorithm for learning threshold functions in the mistake bound model [15] Subsequently, Blum, Hellerstein, and Littlestone [4] considered the following general question: If a class of functions can be learned in polynomial time in a query or mistake bound model, can it be learned by a polynomial time I(n) attribute efficient algorithm in that model (In particular, they considered the cases I(n) log n and I(n) n ff ....

[Article contains additional citation context not shown here]

A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. Journal of Computer and System Sciences, 50(1):32--40, 1995.

....in polynomial time. We show that this does not hold in the randomized membership query model. In the mistake bound model, we consider the problem of learning attribute efficiently using hypotheses that are formulas of small depth. Our results extend the work of Blum et al. [3] and Bshouty et al. 5] 1 Introduction Consider the problem of learning an unknown Boolean function on n variables where n is large. Suppose that the output of this function is completely determined by the values of a fixed set of r of the n variables, where r is small. Thus our real task is to ....

....has only a sublinear dependence on the number of irrelevant attributes (variables) in the target function. Littlestone developed a polynomial time, log n attribute efficient algorithm for learning threshold functions in the mistake bound model [10] Subsequently, Blum, Hellerstein, and Littlestone [3] considered the following general question: If a class of functions can be learned in polynomial time in a query or mistake bound model, can it be learned by a polynomial time I(n) attribute efficient algorithm in that model (In particular, they considered the cases I(n) log n and I(n) n ff ....

[Article contains additional citation context not shown here]

A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. Journal of Computer and System Sciences, 50(1):32--40, 1995.

....an Infinite Attribute Space 9 So, the maximum number of mistakes make by this algorithm is 1 3r(2 log n) One can extend the above technique to learn K DNF formulas by combining the ideas of the above algorithm with those of LEARN K CNF. Recently, however, Blum, Hellerstein, and Littlestone (Blum et al. 1991) have found a method that uses a different approach and achieves similar bounds but with much simpler analysis. Therefore, we shall not present the more complicated method here, and instead refer the reader to that paper for details. 5 Allowing membership queries One natural way to increase the ....

Blum, A., Hellerstein, L., and Littlestone, N. (1991). Learning in the presence of finitely or infinitely many irrelevant attributes. In Proceedings of the Fourth Annual Workshop on Computational Learning Theory, Santa Cruz, California. Morgan Kaufmann.

....to notice is that as learning progresses, the number of specialists in existence maybecome quite large. However, on any individual example, only a small number(number of features choose 2) actually make a prediction. Thus, this setting can be modeled by the infinite attribute model of (Blum, 1992; Blum et al. 1991). 4. Experimental results We ran Winnow and Weighted Majority on 1685 data points from one user and 554 data points from a second user of the CAP system. We presented the examples to the algorithms one by one in chronological order, recording when mistakes were made. Tables 1 and 2 compare the ....

Blum, A., Hellerstein, L., and Littlestone, N. (1991). Learning in the presence of finitely or infinitely many irrelevant attributes. In Proceedings of the Fourth Annual Workshopon Computational Learning Theory, pages 157--166, Santa Cruz, California. Morgan Kaufmann.

....of PAC algorithms are ineffective for polynomial time PAC learning of this class [K93] The general problem of learning in the presence of irrelevant variables has been previously studied in models other than the PAC model. Littlestone [L88] and subsequently Blum, Hellerstein and Littlestone [BHL91], considered the problem in the mistake bound model. Their work focuses on minimizing the effect of irrelevant attributes on the number of mistakes made by a learning algorithm in the mistake bound model. 2 Definitions We consider learning over the boolean domain Xn = f0; 1g n . An example is ....

A. Blum, L. Hellerstein and N. Littlestone. Learning in the presence of finitely or infinitely many attributes. In Proceedings of the Fourth Annual Workshop on Computational Learning Theory, 1991.

....equivalence queries. In particular, the class of monotone DNF formulas is identifiable in polynomial time using membership and equivalence queries, but not with membership queries alone [2] For classes of monotone formulas, constrained instance queries can be simulated using membership queries [8], which implies that the class of monotone DNF formulas cannot be identified in polynomial time using constrained instance queries alone. For the other direction, the techniques of Angluin and Kharitonov [6] can be used to give cryptographic evidence that there are classes of formulas identifiable ....

A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. To appear, Proceedings of the Fourth Annual Workshop on Computational Learning Theory, 1991.

....attribute efficiently in polynomial time. We show that this does not hold in the randomized membership query model. In the mistakebound model, we consider the problem of learning attribute efficiently using hypotheses that are formulas of small depth. Our results extend the work of Blum et al. [4] and Bshouty et al. 7] 1 Introduction Consider the problem of learning an unknown Boolean function on n variables where n is large. Suppose that the output of this function is completely determined by the values of a fixed set of r of the n variables, where r is small. Thus our real task is ....

....which has only a sublinear dependence on the number of irrelevant attributes in the target function. Littlestone developed a polynomial time, log n attribute efficient algorithm for learning threshold functions in the mistake bound model [15] Subsequently, Blum, Hellerstein, and Littlestone [4] considered the following general question: If a class of functions can be learned in polynomial time in a query or mistake bound model, can it be learned by a polynomial time I(n) attribute efficient algorithm in that model (In particular, they considered the cases I(n) log n and I(n) n ff ....

[Article contains additional citation context not shown here]

A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. Journal of Computer and System Sciences, 50(1):32--40, 1995.

....reference, the standard deviation for 554 fair Bernoulli trials is 0.021. may become quite large. However, on any individual example, only a small number (number of features choose 2) actually make a prediction. Thus, this setting can be modeled by the infinite attribute model of (Blum, 1992; Blum et al. 1991). 4 Experimental results We ran Winnow and Weighted Majority on 1685 data points from one user and 554 data points from a second user of the CAP system. We presented the examples to the algorithms one by one in chronological order, recording when mistakes were made. Tables 1 and 2 compare the ....

Blum, A., Hellerstein, L., and Littlestone, N. (1991). Learning in the presence of finitely or infinitely many irrelevant attributes. In Proceedings of the Fourth Annual Workshop on Computational Learning Theory, pages 157--166, Santa Cruz, California. Morgan Kaufmann.

....methods, and several of the results provide bounds on the performance of Bayesian updating, even when the probabilistic assumptions of that approach are not met. Experimental tests of Winnow and related multiplicative methods on natural domains have revealed good behavior (Armstrong et al. 1995; Blum, 1995), and studies with synthetic data show that they scale very well to domains with even thousands of irrelevant features (Littlestone Mesterharm, 1997) More generally, weighting methods are often cast as ways of merging advice from different knowledge sources that may themselves be generated ....

....learning include a bias toward exploring unfamiliar parts of the state space (e.g. Lin, 1992) Both approaches can considerably increase learning rates over random presentations. Most work on selecting and querying unlabeled data has used embedded methods, but Angluin et al. 1993) and Blum et al. 1995) describe theoretical results for a wrapper query method that can be applied to any algorithm. Specifically, they show that when membership queries are available, any algorithm with a polynomial mistake bound for learning a reasonable concept class can be converted in an automated way into one ....

Blum, A., Hellerstein, L., & Littlestone, N. (1995). Learning in the presence of finitely or infinitely many irrelevant attributes. Journal of Computer and System Sciences , 50 , 32--40.

.... algorithm whose performance is especially good when the number of relevant features is small That is, can one devise an algorithm whose sample size or mistake bound has a low dependence on the number of irrelevant features These sorts of algorithms are termed attribute efficient in [ Blum et al. 1991 ] A similar question, addressed by the work of [ Dhagat and Hellerstein, 1994 ] presented at this workshop) is whether one can produce a hypothesis that also itself depends only on a small subset of features. In the framework of computational learning theory, two main techniques have come to ....

.... 1993] One general fact regarding attribute efficient algorithms is that if membership queries are allowed, then any mistake bound algorithm can be converted into one that has only a logarithmic dependence (in terms of number of mistakes plus queries made) on the number of irrelevant features [ Blum et al. 1991 ] Comments It is worth pointing out that in the PAC and mistakebound models (without membership queries) the notion of relevance is really being used as a measure of complexity. In other words, the algorithms do not really care what features are relevant and (generally) they are not really ....

A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. In Proceedings of the Fourth Annual Workshop on Computational Learning Theory, pages 157--166, Santa Cruz, California, August 1991. Morgan Kaufmann.

....is just a reasonableness condition saying that one can take a concept in C defined on n1 variables and embed it into a space with n2 n1 variables and still stay within the class C, and in the reverse direction, one can fix values of some of the variables and still have a legal concept. See [9] for details. 3.5 History The Winnow algorithm was developed by Littlestone in his seminal paper [24] which also gives a variety of extensions and introduces the Mistake Bound learning model. The Mistake Bound model is equivalent to the extended equivalence query model of Angluin [1] and is ....

.... presented for learning decision lists is based on Rivest s algorithm for the PAC model [31] adapted to the Mistake Bound model by Littlestone [25] and Helmbold, Sloan and Warmuth [20] The Infinite Attribute model is defined in Blum [5] and Theorem 9 is from Blum, Hellerstein, and Littlestone [9]. 4 Open Problems 1. Can the bounds of Corollary 3 be achieved and improved with a smooth algorithm The bound of Corollary 3 is achieved using a guess and double algorithm that periodically throws out all it has learned so far and restarts using a new value of fi. It would seem more natural ....

A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. J. Comp. Syst. Sci., 50(1):32--40, 1995.

No context found.

A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. Journal of Computer and System Sciences, 50:32--40, 1995.

No context found.

Avrim Blum, Lisa Hellerstein, and Nick Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. Journal of Computer and System Sciences, 50(1):32--40, February 1995.

No context found.

A. Blum, L. Hellerstein, and N. Littlestone. Learning in the presence of finitely or infinitely many irrelevant attributes. In Proc. 4th Annu. Workshop on Comput. Learning Theory, pages 157--166. Morgan Kaufmann, San Mateo, CA, 1991.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC