T. Linder, G. Lugosi, and K. Zeger, "Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding," IEEE Trans. Inform. Theory, vol. IT-40, pp. 1728--1740, November 1994.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....sample will not be able to distinguish between two sources that are very close. A more detailed qualitative discussion of Theorem 3 will be given in Section 3, where the theorem is proven using the method of types for the direct part. While the problems of universal lossy source coding (e.g. [29, 19, 30, 21, 16, 3, 28]) and of noisy source coding (e.g. 2, 26, 25, 11, 18, 12] have been extensively studied, this work and, in particular, Theorem 3 is the first to address the combination of the two, namely, the problem of universal coding of noisy sources in the probabilistic setting . The merit of Theorem 3, ....

T. Linder, G. Lugosi, and K. Zeger. Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding. IEEE Trans. Inform. Theory, 40:1728--1740, November 1994.

....presenting results on consistency and rate of convergence in Section 2.1.2. 2.1. 2 Consistency and Rate Of Convergence Consistency of the empirical quantizer design under general conditions was proven by Pollard [Pol81, Pol82] The first rate of convergence results were obtained by Linder et al. [LLZ94]. In particular, LLZ94] showed that if the distribution of X is concentrated on a bounded region, there exists a constant c such that ) cd k logn : 8) An extension of this result to distributions with unbounded support is given in [MZ97] Bartlett et al. BLL98] pointed out that the ....

....and rate of convergence in Section 2.1.2. 2.1. 2 Consistency and Rate Of Convergence Consistency of the empirical quantizer design under general conditions was proven by Pollard [Pol81, Pol82] The first rate of convergence results were obtained by Linder et al. LLZ94] In particular, [LLZ94] showed that if the distribution of X is concentrated on a bounded region, there exists a constant c such that ) cd k logn : 8) An extension of this result to distributions with unbounded support is given in [MZ97] Bartlett et al. BLL98] pointed out that the log n factor can be ....

T. Linder, G. Lugosi, and K. Zeger. Rates of convergence in the source coding theorem, in empirical quantizer design and in universal lossy source coding. IEEE Transactions on Information Theory, 40:1728--1740, 1994.

....[33] also bounded the performance of optimum source codes as a function of their blocklength. Recent research by Yang, Zhang, and Wei corrects the work of Pilc and extends it to sources with unknown statistics that possess memory [34] 35] 36] see also related work by Linder, Lugosi, and Zeger [37], 38] Pinkston wrote both a masters thesis [39] and a doctoral dissertation [40] concerning aspects of rate distortion theory. The former concentrated on computing and developing codes for situations in which for certain pairs; this theory parallels analogous in some respects to the theory of ....

T. Linder, G. Lugosi, and K. Zeger, "Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding," IEEE Trans. Inform. Theory, vol. 40, pp. 1728--1740, 1994.

....algorithm with di#erent initial conditions has also proved e#ective in avoiding local optima. We focus on the Lloyd algorithm because of its simplicity, its proven merit at designing codes, and because of the wealth of results regarding its convergence properties [451] 418] 108] 91] 101] [321], 335] 131] 36] The centroid property of optimal reproduction decoders has interesting implications in the special case of a squarederror distortion measure, where it follows easily [137] 60] 193] 184] 196] that . E[q(X) E[X] so that the quantizer output can be considered as ....

....to # k (R) increases with dimension. GRAY AND NEUHOFF: QUANTIZATION 35 Fig. 7. Signal to noise ratios for optimal VQs (dots) and predictions thereof based on the Zador Gersho formula (straight lines) The convergence rate of # k (R)to#(R)asktends to infinity has also been studied [413] 548] [321], 576] Roughly speaking these results show that for memoryless sources the convergence rate is between q log k k and log k k . Unfortunately, this theory does not enable one to actually predict how large the dimension must be in order that # k (R)is within some specified percentage, e.g. ....

[Article contains additional citation context not shown here]

T. Linder, T. Lugosi, and K. Zeger, "Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding," IEEE Trans. Inform. Theory, vol. 40, pp. 1728--1740, Nov. 1994.

....power law form D# R#=A R #b for large R, whence ## R # #=O#n #b=#b 1# #: That is, the overall redundancy per letter follows a power law. Rates of convergence for redundancies of universal quantizers were first reported in [11] and have also recently been examined in [53, 36, 20, 37]. The second problem in two stage universal coding that is solved by regarding it as a quantization problem is the problem of optimal design. That is, given a fixed data length n and first stage length R, what collection of 2 R codes minimizes the expected redundancy in the second stage ....

....or the infinite order redundancy. 7 However, when determining the rate of convergence, we prefer to consider the nth order redundancy over the infinite order redundancy. The rate of convergence of the nth order OPTA to the OPTA is best left as a separate issue of interest in its own right [40, 41, 36]. In the case of noiseless coding, Rissanen and others have provided the optimal rate of convergence of the nth order rate redundancy to zero, when # is a subset of R k [15, 33, 16, 47, 48] The following theorem is a slight generalization of [47, Theorem 1b] which we shall call Rissanen s ....

[Article contains additional citation context not shown here]

T. Linder, G. Lugosi, and K. Zeger. Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding. IEEE Trans. Information Theory, 40:1728--1740, November 1994.

....clear, that a certain number of data points has to be available to reliable estimate a given number of clusters. The question of how many e ective data points are needed to distinguish K clusters has been addressed in the context of uniform convergence of empirical means to their expectations [44, 45]. As a key observation in this context, grouping algorithms should be robust with respect to measurement noise in the image recording process and should not be a ected by modeling uncertainty and the natural within class variability. More speci cally, algorithms should abstract from the ....

.... ) 34) should be small for a given, xed precision parameter . Assume a compact support for P true (x) i.e. Z kxk b P true (x)dx = 1 (35) for some b 0. Then the deviation (34) is uniformly bounded independently of P true (x) by the following theorem due to Linder et al. [45]. Theorem 4 Let N ( 8b) 2 2. Then P R km ( R km ( 4(2N) K(d 1) exp N 2 512b 2 : 36) Thus, the theorem establishes consistency, i.e. the error probability (34) converges to zero for all 0 and K 2 in the limit of in nite data N 1. For a ....

[Article contains additional citation context not shown here]

T. Linder, G. Lugosi, and K. Zeger, \Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding," IEEE Transactions on Information Theory, vol. 40, no. 6, pp. 1728-1740, 1994.

....between supervised learning and unsupervised learning. Both types of learning are mathematically formulated as variational problems with expected loss to be minimized. This observation has been the basis of a number of papers on the uniform convergence behavior in the vector quantization problem [5, 6, 7]. These approaches derive upper and lower bounds for the expected distortion error given a finite number of training points and a box constraint on the support of the data distribution. These bounds do not depend on explicit information of the distribution but have the form of the well known ....

....of relevance by a loss function. Here we assume that such a loss function is given and that only the best hypothesis should be inferred on the basis of empirical data. 2 the derived bound relies mainly on the so called computational temperature and not on the number of clusters as e.g. in [5]. Controlling the complexity of the underlying hypothesis class by the temperature is in fact related to the notion of the optimal margin in supervised learning. Approximating the complexity of such margin classifiers by the VC dimension is similar to approximating the complexity for vector ....

T. Linder, G. Lugosi, and K. Zeger. Rates of convergence in the source coding theorem, in empirical quantizer design and in universal lossy source coding. IEEE Transactions on Information Theory, 40(6):1728--1740, November 1994.

....with different initial conditions has also proved effective in avoiding local optima. We focus on the Lloyd algorithm because of its simplicity, its proven merit at designing codes, and because of the wealth of results regarding its convergence properties [451] 418] 108] 91] 101] [321], 335] 131] 36] The centroid property of optimal reproduction decoders has interesting implications in the special case of a squarederror distortion measure, where it follows easily [137] 60] 193] 184] 196] that ffl E[q(X) E[X] so that the quantizer output can be considered as ....

....increases with dimension. GRAY AND NEUHOFF: QUANTIZATION 35 Fig. 7. Signal to noise ratios for optimal VQs (dots) and predictions thereof based on the Zador Gersho formula (straight lines) The convergence rate of ffi k (R) to ffi (R) as k tends to infinity has also been studied [413] 548] [321], 576] Roughly speaking these results show that for memoryless sources the convergence rate is between q log k k and log k k . Unfortunately, this theory does not enable one to actually predict how large the dimension must be in order that ffi k (R) is within some specified percentage, e.g. ....

[Article contains additional citation context not shown here]

T. Linder, T. Lugosi, and K. Zeger, "Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding," IEEE Trans. Inform. Theory, vol. 40, pp. 1728--1740, Nov. 1994.

....to an underlying distribution P with compact support, i.e. P (kxk 2 Z) 1 and a nearest neighbor rule for M given y. Then bounds for the deviation of the empirical costs H cc ( x i ) y) from the expected costs of a segmentation can be derived independently of the underlying distribution [40]: P (kH cc ( x i ) y) Gamma E [H cc (x; y) k ffl) 4(2N) K(d 1) e GammaN ffl 2 =512Z 2 : 34) Thus for a given accuracy ffl a lower bound for the maximal number of clusters K max distinguishable with probability 1 Gamma ffi given N data points is obtained by K max ffl 2 512B 2 ....

T. Linder, G. Lugosi, and K. Zeger, "Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding," IEEE Transactions on Information Theory, vol. 40, no. 6, pp. 1728--1740, 1994.

....sources over finite alphabets was recently reported by Yang and Zhang in [37] With only a couple of notable exceptions from 1968 (Pilc [24] and Wyner [32] the dual problem of lossy compression at a fixed rate level appears to also have been considered rather recently. Linder, Lugosi and Zeger [19][20] studied various aspects of the distortion redundancy problem and exhibited universal codes with distortion redundancy of order O(log n) Zhang, Yang and Wei [39] proved a lower bound of order O(log n) and they constructed codes achieving this lower bound (to first order) Coding for sources ....

T. Linder, G. Lugosi, and K. Zeger. Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding. IEEE Trans. Inform. Theory, 40(6):1728--1740, 1994.

....reduces at coarser resolution levels, splitting strategy and coarse to fine optimization should be interleaved. The question of how many effective data points are needed to distinguish K clusters has been addressed in the context of uniform convergence of empirical means to their expectations [43, 44]. Assume a sequence of N i.i.d. vectors x i 2 IR d drawn according to an underlying distribution P with compact support, i.e. P (kxk 2 Z) 1 and a nearest neighbor rule for M given y. Then bounds for the deviation of the empirical costs Hkm ( x i ) y) from the expected costs of a ....

....to an underlying distribution P with compact support, i.e. P (kxk 2 Z) 1 and a nearest neighbor rule for M given y. Then bounds for the deviation of the empirical costs Hkm ( x i ) y) from the expected costs of a segmentation can be derived independently of the underlying distribution [44]. For a given accuracy ffl a lower bound for the maximal number of clusters K max distinguishable with probability 1 Gamma ffi given N data points is obtained by Kmax ffl 2 512B 2 (d 1) N log 2N log ffi Gamma log 4 d 1 1 log 2N : 32) J. Puzicha, J.M. Buhmann: Real Time ....

T. Linder, G. Lugosi, and K. Zeger, "Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding," IEEE Transactions on Information Theory, vol. 40, no. 6, pp. 1728--1740, 1994.

....and Gray s [5] vector quantization interpretation of universal lossy source codes is in terms of two pass (or two stage ) weighted universal codes. Another family of two pass lossy compression algorithms is that of empirically designed vector quantizers, discussed by Linder, Lugosi and Zeger [20] among many others. More pointers to the large literature on vector quantization can be found in the recent review paper by Gray and Neuhoff [14] Preliminary results from a work closer in spirit to our approach were recently reported by Zamir and Rose in [37] 38] We analyze the performance of ....

T. Linder, G. Lugosi, and K. Zeger. Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding. IEEE Trans. Inform. Theory, 40(6):1728--1740, 1994.

....= n i=1 Omega i into n cells Omega i and to require constant assignments of data to clusters within a cell, i.e. 8x 2 Omega i m(x) m i ; 1 i n. Other learning theoretical approaches to k means clustering assume that data are always assigned to the closest mean [Pollard, 1982, Linder et al. 1994, Devroye et al. 1996, Linder et al. 1997] An analysis without this constraint on data assignments can be found in [Kearns et al. 1997] i.e. algorithm III with hard assignments selected according to predefined probabilities ressembles a randomized selection of a loss function from the ....

Linder, T., Lugosi, G., and Zeger, K. (1994). Rates of convergence in the source coding theorem, in empirical quantizer design and in universal lossy source coding. IEEE Transactions on Information Theory, 40(6):1728--1740.

....error. This is another instance in which information theoretic and statistical considerations lead to different solutions due to the different loss functions employed. VI Historical Remarks A theoretical study of the training sequence method for lossy code design has recently been accomplished [8]. Minimax and maximin coding are due to Davisson [4] For a sufficiently smooth parametric model class with finitely many parameters (such as a Markov model class of finite order) it is known that the minimax redundancy for n data samples grows like k 2 log 2 n, where k is the number of ....

T. Linder, G. Lugosi, and K. Zeger, "Rates of Convergence in the Source Coding Theorem, in Empirical Quantizer Design, and in Universal Lossy Source Coding," IEEE Trans. Inform. Theory, vol. 40, pp. 1728-1740, 1994

.... What is the minimum value of N such that for every source, the rate distortion performance could be essentially as good as that of the optimal rate R, l dimensional quantizer In fact, one half of this question, the sufficiency part, has been answered recently by Linder, Lugosi, and Zeger [4]. Their results imply that if N is exponentially larger than M (i.e. N 2 (R ffi)l for some ffi 0) and the N side information bits represent a sequence of (finely quantized) i.i.d. l dimensional training vectors, 1 then the distortion is essentially as small as D l (R) the minimum ....

....theorem requires independent training vectors, which are never quite available from any finite length sample unless the stationary ergodic source is memoryless. For memoryless sources, however, the necessary amount of training vectors need not grow with l because it is entirely dictated by 1 In [4] the training vectors were not quantized. Nevertheless, in order to represent the training vectors by a finite number of bits as in our setting, we think of them as being quantized. This quantization, however, should be sufficiently fine so that the resulting additional distortion will be ....

[Article contains additional citation context not shown here]

T. Linder, G. Lugosi and K. Zeger, "Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding," IEEE Trans. Inform. Theory, vol. IT-40, no. 6, pp. 1728-1740, November 1994.

No context found.

T. Linder, G. Lugosi, and K. Zeger, "Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding," IEEE Trans. Inform. Theory, vol. 40, pp. 1728--1740, Nov. 1994.

No context found.

T. Linder, G. Lugosi, and K. Zeger, "Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding," IEEE Trans. Inform. Theory, vol. 40, pp. 1728--1740, Nov. 1994.

No context found.

T. Linder, G. Lugosi, and K. Zeger. Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding. IEEE Trans. Inform. Theory, 40:1728--1740, Nov. 1994.

....needs to study the dependence of D(Q n ) on finite n. Assume that the training set consists of n independent sample vectors drawn from the source distribution and let E[D(Q n ) denote the expected value (taken over the training sequence) of the mean squared test distortion D(Q n ) In [4] it was shown that for all source distributions supported by a given bounded region, the test distortion of the empirically optimal quantizer satisfies E[D(Q n ) Gamma D(Q ) c n Gamma1=2 for some positive constant c. This upper bound was shown to have the right order in a minimax ....

....of the training ratio fi = n=k (where k 3 is the codebook size) by showing that there exist source distributions for which E[D n (Q n ) D(Q ) 1 Gamma c 0 p fi (3) where c 0 0 is a universal constant. Theorem 3 presents an improved, explicit form of an earlier result in [4] to show that the lower bound E[D n (Q n ) D(Q ) Gamma c p n holds for a c 0, uniformly for all sources supported on a given bounded set. This shows that bound (2) is tight in the sense that only the constants may be improved. The proofs of these results are given in Section 4. ....

[Article contains additional citation context not shown here]

T. Linder, G. Lugosi, and K. Zeger, "Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding," IEEE Trans. Inform. Theory, vol. 40, pp. 1728--1740, Nov. 1994.

.... known source statistics The consistency of design based on global minimization of the empirical distortion was established with various levels of generality by Pollard [5] Abaya and Wise [6] and Sabin [7] The finite sample performance was also analyzed by Pollard [8] Linder, Lugosi, and Zeger [9], and Chou [10] The consistency of the generalized Lloyd algorithm was also established by Sabin [7] and Sabin and Gray [11] An interesting interpretation of the quantizer design problem was given by Merhav and Ziv [12] who obtained lower bounds on the amount of side information Manuscript ....

....Theorem 1 proves that the expected squared error distortion of the quantizer minimizing the appropriately defined empirical distortion over training vectors is within of the distortion of the quantizer which is optimal for the given source and channel. This is the same rate as that obtained in [9] for the standard vector quantizer problem. In Section IV, empirical design for sources corrupted by additive noise is considered. A method is presented which combines nonparametric estimation with empirical error minimization. Theorem 2 proves that if the conditional mean of the clean source ....

[Article contains additional citation context not shown here]

T. Linder, G. Lugosi, and K. Zeger, "Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding," IEEE Trans. Inform. Theory, vol. 40, no. 6, pp. 1728--1740, Nov. 1994.

....infinity. Of course mere consistency does not give any indication how large the training data should be so that the distortion of the designed quantizer be close to the optimum. This question can only be answered by analyzing the finite sample behavior of D n . In this direction, it was shown in [10, 15] that there exists a c such that D n Gamma D c q log n=n for all sources over a bounded region. This result has since been extended to empirical quantizer design for vector quantizers operating on noisy sources and for vector quantizers for noisy channels [11] An extension to unbounded ....

....optimal quantizer, denoted Q n , is an empirically designed quantizer which minimizes the empirical error D n (Q) 1 n n X i=1 kx i Gamma Q(x i )k 2 over all k point nearest neighbor quantizers Q. The first result upper bounding the minimax distortion redundancy was given in [10], where it was proved that for the empirically optimal quantizer J(Q n ; cd 3=2 s k log n n (2) for all , where c is a universal constant. The main message of the above inequality is that there exists a sequence of empirical quantizers such that for all distributions supported on a ....

[Article contains additional citation context not shown here]

T. Linder, G. Lugosi, and K. Zeger. Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding. IEEE Trans. Inform. Theory, 40:1728--1740, Nov. 1994.

....error Dn (Q) 1 n n X i=1 kX i Gamma Q(X i )k 2 over all k point nearest neighbor quantizers Q. One can prove by using standard uniform large deviation inequalities that J(Q n ; cd 3=2 r k log n n (1) for all , where c is a universal constant (see Linder, Lugosi, and Zeger [6], 7] The main message of the above inequality is that there exists a sequence of empirical quantizers such that for all distributions supported on a given d dimensional sphere the expected distortion redundancy decreases as O( p log n=n) With analysis based on an inequality of Alexander ....

....decreases as O( p log n=n) With analysis based on an inequality of Alexander [1] it is possible to get rid of the p log n factor. More precisely, one can prove that J(Q n ; c 0 d 3=2 r k log(kd) n (2) for all , where c 0 is another universal constant (see the discussion in [6] and Problem 12.10 in [4] It has been an open question how tight these upper bounds are. Some have conjectured that J(Q n ; in fact, decreases at a faster, O(1=n) rate. For example, for one codepoint quantizers (i.e. when k = 1) it is easy to see that J(Qn ; O(1=n) if the single ....

T. Linder, G. Lugosi, and K. Zeger. Rates of convergence in the source coding theorem, empirical quantizer design, and universal lossy source coding. IEEE Transactions on Information Theory, 40:1728--1740, 1994.

....error D n (Q) 1 n n X i=1 kX i Gamma Q(X i )k 2 over all k point nearest neighbor quantizers Q. One can prove by using standard uniform large deviation inequalities that J(Q n ; cd 3=2 s k log n n (1) for all , where c is a universal constant (see Linder, Lugosi, and Zeger [6], 7] The main message of the above inequality is that there exists an empirical quantizer such that for all distributions supported on a given d dimensional sphere the expected distortion decreases as O( q log n=n) With analysis based on an inequality of Alexander [1] it is possible to get ....

....decreases as O( q log n=n) With analysis based on an inequality of Alexander [1] it is possible to get rid of the p log n factor. More precisely, one can prove that J(Q n ; c 0 d 3=2 s k log(kd) n (2) for all , where c 0 is another universal constant (see the discussion in [6] and Problem 12.10 in [4] It has been an open question how tight these upper bounds are. Some have conjectured that J(Q n ; in fact, decreases at a faster, O(1=n) rate. For example, for one codepoint quantizers (i.e. when k = 1) it is easy to see that J(Q n ; O(1=n) if the single ....

T. Linder, G. Lugosi, and K. Zeger. Rates of convergence in the source coding theorem, empirical quantizer design, and universal lossy source coding. IEEE Transactions on Information Theory, 40:1728--1740, 1994.

No context found.

T. Linder, G. Lugosi, and K. Zeger, "Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding," IEEE Trans. Inform. Theory, vol. IT-40, pp. 1728--1740, November 1994.

No context found.

T. Linder, G. Lugosi, and K. Zeger. Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding. IEEE Trans. Inform. Theory, 40:1728--1740, November 1994.

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC