Leung-Yan-Chrong, S. K. and Cover, T. M. (1978) Some equivalences between Shannon entropy and Kolmogorov complexity. IEEE. Trans. Inform. Theory, IT-24, 331--338.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....k y k = y =s(y) The above bound for 2 s(y) can be rewritten as log P(y) 1 s(y) # log P(y) 2. Then KP(y) # KP B (y) log P(y) # Now we can show that the average code word KP(x n) is equal to the Shannon entropy of the corresponding probability distribution [34] and [3] For any n let Q n be a computable probability distribution in # n . The computability means that there is an algorithm computing Q n (x) given n and a finite sequence x # # n with arbitrary degree of accuracy. The mathematical expectation of a real function f (x) on # n with ....

Leung-Yan-Chrong, S. K. and Cover, T. M. (1978) Some equivalences between Shannon entropy and Kolmogorov complexity. IEEE. Trans. Inform. Theory, IT-24, 331--338.

....: By (9) we thus obtain Delta k n k 4d k=2 jM n ( ff )j 1=2 N ff Delta k n (1 4 q d= k nk;ff ) Delta Gamma1 n ) k k 4d k=2 jM n ( ff )j 1=2 : 10) For encoding ff we need basically a code for the natural numbers. We use the socalled universal code ( 1] 2] [6], 8] It has the length L(n) log (n) const where const 2:87 is determined by P 2 GammaL(n) 1. Since we need an additional bit for the sign of ff i we obtain as the code length for ff k X i=1 log (jff i j) k(const 1) From the definition of C ff , it follows that ....

S.K. Leung-Yan-Cheong and T. Cover, "Some equivalences between Shannon entropy and Kolmogorov complexity," IEEE Trans. Inform. Theory 24, 331-338, 1978.

.... been used in psychology (Attneave, 1959) Garner, 1962) Staniland, 1966) is to further decompose the term E(D) Gamma X a1 ; a n P (a 1 ; an ) log 2 P (a 1 ; an ) 7 The Shannon measure of complexity was related to measures based on Kolmogorov and Chaitin measures by (Leung Yan Cheong and Cover, 1978) and later by Abu Mostafa (1986) who showed that they were equivalent for a wide range of problems. 8 It is interesting to note that the study of Lempel and Ziv concluded in the now famous LZ compression algorithm (Ziv and Lempel, 1978) a variant of which is used for the program compress in ....

S.K. Leung-Yan Cheong and T.M. Cover. Some equivalences between Shannon entropy and Kolmogorov complexity. IEEE Transactions on Information Theory, IT-24:331--338, 1978. 160

....1. M (ff) ae K(ff) 2. DH(M (ff) ff; and 3. DH(K(ff) ff. The first item is clear because it is known that there exists a finite state machine that compresses each x 1 1 2 M (ff) at the rate no more than ff and any finite state machine can be expressed by an algorithm. For the second, [2,3] gave the derivation of c(I) DH(I) where c(I) inf c( I) c( I) sup x 1 1 2I c( x 1 1 ) c( x 1 1 ) lim inf n 1 (n; x 1 1 ) n ; n; x 1 1 ) is the minimum length among prefixies of (x 1 1 ) that enables to decode x n 1 uniquely, and is any mapping I ....

....1 1 ) that enables to decode x n 1 uniquely, and is any mapping I B 1 [3] Then, DH(M (ff) ff holds because c(M (ff) ff is achieved by Lempel Ziv coding [ The reverse inequality follows from the fact that there exists I ae M (ff) such that DH( I) ff. See [1] For the last, [2,3] also gave the derivation of c T (I) sup x 1 1 2I lim inf n 1 KC(x n 1 ) n, where c T (I) inf OE c( I) and the infimum ranges over the mappings OE : I B 1 that are realized by algorighms. Then, clearly ff = c T (K(ff) c(K(ff) DH(K(ff) The reverse inequality follows ....

T. M. Cover and S. K. Leung, "Some equivalences between Shannon entropy and Kolmogorov complexity", IEEE Trans. Inform Theory, IT-27: 292-298, 1981.

.... is a code that assigns to each source letter a different codeword (notice that in our definition of code, any code is a one to one code) We remark that in the literature one to one codes have been studied in two different frameworks depending on whether the empty codeword is used [3, 5] or not [5, 8, 9, 12]. A labeled (binary) tree is a (binary) tree in which each edge is labeled with 0 or with 1 and, if a node has two children, the two edges going from that node to its two children have different labels. A node in a labeled tree represents the codeword given by the sequence of labels in the path ....

S.K. Leung-Yan-Cheong and T.M. Cover, Some equivalences between Shannon entropy and Kolmogorov complexity, IEEE Trans. on Information Theory, 24 (May 1978), pp. 331--338.

....n ; 0.1) and we compare the functions (F ) supf(fi) fi 2 Fg and (F ) supf(fi) fi 2 Fg (0.2) defined for languages to the corresponding entropy H F or Hausdorff dimension dimF . Since we are mainly interested in the above mentioned first order approximations, the established in [KS] [LC] and [Sc2] relations between Kolmogorov, Chaitin and other concepts of program complexity prove that the functions and do not depend on the particular kind of complexity we use. We therefore (also in view of Theorem 2.5 and Proposition 2.10 below) agree on the following concept of conditional ....

Leung-Yan-Cheong, S.K. and Cover, T., Some equivalences between Shannon entropy and Kolmogorov complexity. IEEE Trans. Inform. Theory IT24 (1978), 331 - 338.

....1 ; x 2 ; Delta Delta Delta ; xn ) is a number , whereas the complexity, which we now write C(x 1 ; x 2 ; Delta Delta Delta ; xn ) is a stochastic variable. Taking the expectation of the complexity should bring us close to the entropy. And indeed, it was shown by Leung Yan Cheong and Cover [9] that, under a computability condition on the marginal probability distributions, there is a constant k such that for all n H(x 1 ; Delta Delta Delta ; xn ) E C(x 1 ; Delta Delta Delta ; xn ) H(x 1 ; Delta Delta Delta ; xn ) k: In particular, for the entropy rate of the process one ....

S.K. Leung-Yan-Cheong and T.M. Cover (1978). Some equivalences between Shannon entropy and Kolmogorov complexity. IEEE Trans. Information Th. IT-24, 331--338.

....entropy is H S , then L H S : Let L 1:1 be the average codeword length of a one to one code for a source S of m letters whose entropy is H S . The following bound is due to Rissanen [7] L 1:1 H S Gamma log log m: 3) We will use also the following bound due to Leung Van Cheong and Cover [6], L 1:1 H S Gamma 2 log(H S 2) 4) In Section 4 we utilize bounds better than (3) and (4) to improve our results. However, for the sake of simplicity in deriving the bounds we utilize (3) and (4) Finally, we recall the following results. kraft s equality. In any binary search tree we have ....

S.K. Leung-Yan-Cheong and T.M. Cover, Some equivalences between Shannon entropy and Kolmogorv complexity, IEEE Trans. Inf. Theory, 24 (May 1978), pp. 331--338.

....: g: Clearly, the best f0;1g encoding uses the N shortest codewords of the codebook. It is easy to see that the length of the i th shortest codeword is n i = log i 2 1 : 8) We denote by L the expected length of the best f0;1g encoding. Leung Yan Cheong and Cover [2] proved the following bound L H Gamma log (H 1) Gamma 6 (9) where log x 4 = log x log log x Delta Delta Delta stopping at last positive term. Alon and Orlitsky s bound (3) improves on previous bound (any lower bound for L ffl holds for L as well) Verriest [4] proved that ....

S. K. Leung-Yan-Cheong and T. M. Cover, "Some Equivalences Between Shannon Entropy and Kolmogorov Complexity", IEEE Trans. Inform. Theory, vol. IT--24, no. 3, pp. 331--338, May 1978.

No context found.

T.M. Cover and S.K. Leung, \Some equivalences between Shannon entropy and Kolmogorov complexity," IEEE Trans. Inf. Theory, vol.24, pp.331-338, 1978.

No context found.

Leung-Yan-Cheong, Sik K., and Thomas M. Cover, Some equivalences between Shannon entropy and Komolgorov complexity, IEEE Transactions on Information Theory, Vol. IT-24, No. 3, May 1978, pp 331 - 338.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC