S. Kullback and R. A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22:79-- 86, 1951.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....The relative information measure (often called mutual information ) H(p, q) of two continuous PDFs p(x) and q(x) is defined as: H(p, q) p(x) ln p(x) q(x) dx. 15) 16 This scalar is also called the Kullback Leibler divergence, after the duo that first presented it, 22] [23]) or also mutual entropy, or cross entropy, of both probability measures p(x) and m(x) 22, 28] It is a (coordinate independent) measure for how much information one needs to go from the probability distribution m(x) to the probability distribution p(x) As Shannon s entropy, H p(x) m(x) ....

S. Kullback and R. A. Leibler. On information and su#ciency. The Annals of Mathematical Statistics, 22:79--86, 1951.

....a number of theoretical results about the proposed measure and then present two of its key applications. On the theoretical side, we prove that our proposed measure satisfies the three properties of distance. We also contrast our distance measure with classical measures, including KL divergence [12] where we present some results on its ability to bound belief changes. Specifically, we show that belief change between two states of belief can be unbounded, even when their KL divergence tends to zero. We show, however, that KL divergence can be used to bound the average change in beliefs as ....

....however, that KL divergence can be used to provide an average case bound on belief changes and we also provide a relationship between that bound and ours. We start first with Kullback Leibler (KL) divergence, which is one of the most common measures for comparing probability distributions [12]. Definition 3.1 Let P r and P r # be two probability distributions over the same set of worlds w. The KL divergence between P r and P r # is defined as: P r(w) ln . The first thing to note about KL divergence is that it is incomparable with our distance measure. Example 3.1 ....

S. Kullback and R. A. Leibler. On Information and Su#ciency. Annals of Mathematical Statistics, volume 22, pages 79-86, 1951.

....d 100 and ED [g(X; f i (X) C g 1 i (2) for every i = 1; 2; d. where C g is a constant that is independent of the target concept f i . We call C g the inherent bias of g. 3. 2 The KL divergence We present a de nition of the KL divergence, or Kullback Leibler divergence [KL51]. De nition 4 (KL divergence) For 2 random variables P , Q with identical support, we de ne their KL divergence to be KL(P jjQ) P (x) log : 3) The KL divergence is also known as the relative entropy. The reader is referred to [KL51, K59, B95, CT91] for a comprehensive treatise on ....

....the KL divergence, or Kullback Leibler divergence [KL51] De nition 4 (KL divergence) For 2 random variables P , Q with identical support, we de ne their KL divergence to be KL(P jjQ) P (x) log : 3) The KL divergence is also known as the relative entropy. The reader is referred to [KL51, K59, B95, CT91] for a comprehensive treatise on the KL divergence. We state several properties of the KL divergence and postpone their proofs to Appendix A. These properties are highly unlikely to be original. We list them here only for the completeness of the paper and for the readers convenience. Lemma 2 ....

S. Kullback and R. A. Leibler, On Information and Suciency, Annals of Mathematical Statistics 22, pp. 79-86, 1951.

....cues is assumed to give the best estimate of the true PDF P (e t t ) across the state space . So the performance of the j cue can be quantified by measuring how closely the cue s PDF P (e j,t t ) matches t ) This can be done using the relative entropy, or the Kullback Leibler distance [6], an information theoretic measure of how accurate an approximation one PDF is to another, given by # t (P (e t t ) s t t ) log P (e j,t where s t are the particle states at time t. Soto and Khosla [8] used this metric to rate the performance of their cues, and Triesh and ....

S. Kullback and R. A. Leibler. On information and sufficiency. Annals of Mathematical Statistics, 22(1):79--86, March 1951.

.... The approximation should contain the same normalization factor, i.e. x2U I T (x) x2U I OE(x) If we denote by p T and p OE the probability distributions proportional to T and OE respectively, the distance from a tree T to a potential OE is measured by Kullback Leibler s cross entropy [32] of p T and p OE : D(OE; T ) D(p OE ; p T ) Gamma x I 2U I p OE (x I ) log p OE (x I ) p T (x I ) 7) Proposition 1 [6] Let OE be a potential over a set of variables X I , and J I. If a tree T is such that every leaf T contains the value y J =x OE(y) jU I GammaJ j, then T ....

S. Kullback and R. A. Leibler. On information and sufficiency. Annals of Mathematical Statistics, 22:76--86, 1951.

....standard model, the learned CPTs of N may have other domains than the CPTs of . Hence a global measure for the di erence between the gold standard model and the estimated model is required. In the tests performed, we have measured this di erence by using the Kullback Leibler (KL) divergence [37] between the gold standard model and the estimated model. The KL divergence is de ned as D f jj f N x f(xj ) log f(xj ) 15) There are many arguments for using this particular measurement for calculating the quality of the approximation, see [10] One of them is the fact ....

Solomon Kullback and Richard A. Leibler. On information and suciency. Annals of Mathematical Statistics, 22:79-86, 1951.

....across sensors. 2.1 Mutual Information Mutual information is a quantity characterizing the statistical dependence between two random variables. Although most widely known for its application to communications (see e.g. 1] here it arises in the context of discrimination and hypothesis testing [2]. Correlation is equivalent to mutual information only for jointly Gaussian random variables. The common assumption of Gaussian distributions and its computational e#ciency have given it wide applicability to association problems. However, there are many forms of dependency which are not captured ....

S. Kullback and R. A. Leibler. On information and su#ciency. Annals of Mathematical Statistics, 22:79--86, 1951.

....reals with , which will be called configuration of the rule set . Each corresponds to a rule . The configuration k decides about the quantities of dissimilarity we derive from . The Kullback Leibler divergence generally measures the dissimilarity between two probability mass functions [6] and was applied successfully to statistical language modelling and prediction problems [7] The Kullback Leibler D(x,y) divergence for two words x, y is defined as In the basic version of the Kullback Leibler divergence, which is expressed by formula (1) w is a linguistic property and is the ....

Kullback, S. and Leibler, R.A.: On Information and Sufficiency, Annals of MAthematical Statistics 22, 1951

....and is defined as the difference between the information held in the actual distribution and the information held in the approximating distribution. By subtracting these two quantities as in 16 Equation 2.4, we end up with the measure given Vx)log Vx) Vx)log Va(x) 2.4) in Equation 2. 5 [13]. V(x) 2.5) I(V,V, V(x) log V(x This quantity is equal to 0 if and only if P(x) P(x) for all x. In all other cases, it is positive, growing in value as the two distributions increase their divergence. This distance measure is one key to developing a method to construct a dependence tree. ....

S. Kullback and R. A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 1951.

....The BCD method first transforms each time series into a Markov chain, with its dynamics represented simply by a transition probability matrix. Next, it goes through an agglomerative procedure by trying to merge the two closest Markov chains at each step, using the Kullback Leibler divergence [13] as the dissimilarity (cf. distance) measure between transition probability matrices. Based on a greedy heuristic search approach, this procedure continues until the resulting model is found to be less probable than the model before merging. Thus the number of clusters can be determined ....

S. Kullback and R.A. Leibler. On information and su#ciency. Annals of Mathematical Statistics, 22:79--86, 1951.

....of AS(H E) Smyth and Goodman [28] defined the information content of rules. For E J(H AS(H = G(E) AS(H log IND(E,H) AS(H log IND(H,E) a b) b d) 20) This measure is closely related to the divergence measure proposed by Kullback and Leibler [12]. 3.3 Two way support The measure of independence IND has been used by many authors. Silverstein et al. 27] referred to it as a measure of interest. Buchter and Wirth [3] regarded it as a measure of dependence. Gray and Orlowska [10] used the same measure, and provided the interpretation given ....

Kullback, S. and Leibler, R.A. On information and su#ciency, Annals of Mathematical Statistics, 22, 79-86, 1951.

....0.15 0.11 0.15 0.60 0.68 0.46 0.63 The Bhattacharyya distance [20] provides one measure of similarity. Given two distributions Pon and Poff over the variable y, the Bhattacharyya distance between them is 5B(Pon,Poff) log f y Pon(Y)Poff(Y)dy. 3) Alternatively, the Kullback Leibler divergence [22] between Pon and Poff is given by 5KL(Pon,Poff) Port(Y) log Pon(Y) dy. 4) po(y) Note, the Kullback Leibler divergence is asymmetric, which strictly means that it is not a distance metric, but it still provides a measure of the difference between the two distributions. Below we use these ....

Kullback, S. and R. A. Leibler: 1951, 'On information and sufficiency'. Annals of Mathematical Statistics 22, 79-86.

....that we can characterize a Markov distribution by an entropy function. We now outline how we learn the dependency structure of a Markov distribution. Our method for learning the dependency structure of a Markov distribution is based on minimizing the Kullback Leibler I(OE; OE cross entropy [3], a measure of closeness between a true distribution OE and an approximate distribution OE . The cross entropy is defined by: OE(x) log OE(x) x) where x is a configuration (tuple) of the set of attributes R = fA 1 ; Am g. With a fixed OE, we can choose, from the set ....

S. Kullback and R. Leibler. Information and sufficiency. Annals of Mathematical Statistics, 22:79--86, 1951.

....similarity. Two attributes (two columns in data matrix X) with exactly the same probability distributions are identical for the purpose of data mining and so one can be deleted. Attributes that have probability distributions that are close in terms of their Kullback Leibler (KL) distance (see [KL51]) can still be grouped together without much of an impact. In addition, a natural derived attribute, the mixed distribution (a normalized sum of two columns) is now available to represent the group. This process can be generalized. The grouping simplifies the original matrix X to the compressed ....

Kullback, S. and Leibler, R.A. On information and sufficiency. Annals of Mathematical Statistics, 22, 76-86, 1951.

No context found.

S. Kullback and Leibler. On information and suffiency. Annals of Mathematical Statistics, 22:76--86, 1951.

No context found.

S. Kullback and R. A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22:79-- 86, 1951.

No context found.

S. Kullback and R.A. Leibler. On information and su#ciency. Annals of Mathematical Statistics, 22:79--86, 1951.

No context found.

Solomon Kullback and Richard A. Leibler. On information and sufficiency. Annals of Mathematical Statistics, 22:79--86, 1951.

No context found.

Kullback S. and Leibler. On information and sufficiency. Annals of Mathematical Statistics, (22):79--86, 1951.

No context found.

S. Kullback and R. A. Leibler. On information and suciency. Annals of Mathematical Statistics, 22:7986, 1951. 6

No context found.

S. Kullback and R. A. Leibler. On information and su#ciency. Annals of Mathematical Statistics, 22:79--86, 1951. 34

No context found.

S. Kullback and R.A. Leibler. On information and sufficiency. Annals of Mathematical Statistics, 22(1):79--86, Mar 1951.

No context found.

Kullback, S., Leibler, R.A.: On information and sufficiency. Annals of Mathematical Statistics, vol. 22, pp. 76-86, 1951.

No context found.

S. Kullback and R. Leibler. On information and su#ciency. Annals of Mathematical Statistics, 22:79--86, 1951.

No context found.

S. Kullback and R. Leibler, \On information and suciency," Annals of Mathematical Statistics, vol. 22, 1951.

First 50 documents  Next 50

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC