We characterize the best achievable performance of lossy compression algorithms operating on arbitrary random sources, and with respect to general distortion measures. Direct and converse coding theorems are given for variable-rate codes operating at a xed distortion level, emphasizing: (a) non-asymptotic results, (b) optimal or near-optimal redundancy bounds, and (c) results with probability one. This development is based in part on the observation that there is a precise correspondence between compression algorithms and probability measures on the reproduction alphabet. This is analogous to the Kraft inequality in lossless data compression. In the case of stationary ergodic sources our results reduce to the classical coding theorems. As an application of these general results, we examine the performance of codes based on mixture codebooks for discrete memoryless sources. A mixture codebook (or Bayesian codebook) is a random codebook generated from a mixture over some class of reproduction distributions. We demonstrate the existence of universal mixture codebooks, and show that it is possible to universally encode memoryless sources with redundancy of approximately (d=2) log n bits, where d is the dimension of the simplex of probability distributions on the reproduction alphabet.

In lossless data compression, given a sequence of observations (Xn)n≥1 and a family of probability distributions {Qθ}θ∈Θ, the estimators ( ˜ θn)n≥1 obtained by minimizing the ideal Shannon code-lengths over the family {Qθ}θ∈Θ, ˜θn: = arg min − log Qθ(X θ∈Θ n � 1), where X n 1: = (X1, X2,..., Xn), coincide with the classical maximum-likelihood esti-mators (MLEs). In the corresponding lossy compression setting, the ideal Shannon code-lengths are approximately − log Qθ(B(X n 1, D)) bits, where B(Xn 1, D) is the distortion-ball of radius D around the source sequence Xn 1. In this work we consider the analogous estimators obtained by minimizing these lossy code-lengths, ˆθn: = arg min − log Qθ(B(X θ∈Θ n � 1, D)). The ˆ θn are a lossy version of the MLEs, which we call “lossy MLEs”. We investigate the strong consistency of lossy MLEs when the Qθ are i.i.d. and the sequence (Xn)n≥1 is stationary and ergodic. 1

Abstract—We propose a scheme for lossy compression of discrete memoryless sources: The compressor is the decoder of a nonlinear channel code, constructed from a sparse graph. We prove asymptotic optimality of the scheme for any separable (letter-by-letter) bounded distortion criterion. We also present a suboptimal compression algorithm, which exhibits near-optimal performance for moderate block lengths. Index Terms—Discrete memoryless sources, lossy data compression, rate–distortion theory, source–channel coding duality, sparse-graph codes. I.

lossy compression

We consider the problem of lossy data compression for data arranged on twodimensional arrays (such as images), or more generally on higher-dimensional arrays (such as video sequences). Several of the most commonly used algorithms are based on pattern matching: Given a distortion level D and a block of data to be compressed, the encoder rst nds a D- close match of this block into some database, and then describes the position of the match. We consider two idealized versions of this scenario. In the rst one, the database is taken to be a collection of independent realizations of the same size and from the same distribution as the original data. In the second, the database is assumed to be a single long realization from the same source as the data. We show that the compression rate achieved (in either version) is no worse than R(D=2) bits per symbol, where R(D) is the rate-distortion function. This is proved under the assumption that the data is generated by a Gibbs distribution, and it generalizes the corresponding one-dimensional bound of Steinberg and Gutman. Using recent large deviations results by Dembo and Kontoyiannis and by Chi, we are able to give short proofs for the present results.

We give a development of the theory of lossy data compression from the point of view of statistics. This is partly motivated by the enormous success of the statistical approach in lossless compression, in particular Rissanen’s celebrated Minimum Description Length (MDL) principle. A precise characterization of the fundamental limits of compression performance is given, for arbitrary data sources and with respect to general distortion measures. The starting point for this development is the observation that there is a precise correspondence between compression algorithms and probability distributions (in analogy with the Kraft inequality in lossless compression). This leads us to formulate a version of the MDL principle for lossy data compression. We discuss the consequences of the lossy MDL principle and explain how it leads to potential practical design lessons for vector-quantizer design. We introduce two methods for selecting efficient compression algorithms, the lossy Maximum Likelihood Estimate (LMLE) and the lossy Minimum Description Length Estimate (LMDLE). We describe their theoretical performance and give examples illustrating how the LMDLE has superior performance to the LMLE. 1

Abstract — We introduce a universal quantization scheme based on random coding, and we analyze its performance. This scheme consists of a source-independent random codebook (typically mismatched to the source distribution), followed by optimal entropy-coding that is matched to the quantized codeword distribution. A single-letter formula is derived for the rate achieved by this scheme at a given distortion, in the limit of large codebook dimension. The rate reduction due to entropy-coding is quantified, and it is shown that it can be arbitrarily large. In the special case of “almost uniform ” codebooks (e.g., an i.i.d. Gaussian codebook with large variance) and difference distortion measures, a novel connection is drawn between the compression achieved by the present scheme and the performance of “universal ” entropy-coded dithered lattice quantizers. This connection generalizes the “half-a-bit ” bound on the redundancy of dithered lattice quantizers. Moreover, it demonstrates a strong notion of universality where a single “almost uniform ” codebook is near-optimal for any source and any difference distortion measure. The proofs are based on the fact that the limiting empirical distribution of the first matching codeword in a random codebook can be precisely identified. This is done using elaborate large deviations techniques, that allow the derivation of a new “almost sure” version of the conditional limit theorem.

We study the asymptotics related to the following matching criteria for two independent realizations of point processes X ∼ X and Y ∼ Y. Given l> 0, X ∩[0,l) serves as a template. For each t> 0, the matching score between the template and Y ∩ [t,t + l) is a weighted sum of the Euclidean distances from y − t to the template over all y ∈ Y ∩ [t,t + l). The template matching criteria are used in neuroscience to detect neural activity with certain patterns. We first consider Wl(θ), the waiting time until the matching score is above a given threshold θ. We show that whether the score is scalar- or vector-valued, (1/l)log Wl(θ) converges almost surely to a constant whose explicit form is available, when X is a stationary ergodic process and Y is a homogeneous Poisson point process. Second, as l → ∞, a strong approximation for −log[Pr{Wl(θ) = 0}] by its rate function is established, and in the case where X is sufficiently mixing, the rates, after being centered and normalized by √ l, satisfy a central limit theorem and almost sure invariance principle. The explicit form of the variance of the normal distribution is given for the case where X is a homogeneous Poisson process as well.

We investigate the Gaussian small ball probabilities with random centers, find their deterministic a.s.-equivalents and establish a relation to infinite-dimensional high-resolution quantization. 1. Introduction. Consider

Constrained sequences are strings satisfying certain additional structural restrictions (e.g., some patterns are forbidden). They find applications in communication, digital recording, and biology. In this paper, we restrict our attention to the so-called (d, k) constrained binary sequences in which any run of zeros must be of length at least d and at most k, where 0 ≤ d < k. In many applications one needs to know the number of occurrences of a given pattern w in such sequences, for which we coin the term constrained pattern matching. For a given word w, we first estimate the mean and the variance of the number of occurrences of w in a (d, k) sequence generated by a memoryless source. Then we present the central limit theorem and large deviations results. As a by-product, we enumerate asymptotically the number of (d, k) sequences with exactly r occurrences of w, and compute Shannon entropy of (d, k) sequences with a given number of occurrences of w. We also apply our results to detect under- and overrepresented patterns in neuronal data (spike trains), which satisfy structural constraints that match the framework of (d, k) binary sequences. Throughout this paper we use techniques of analytic algorithmics such as combinatorial calculus, generating functions, and complex asymptotics. Categories and Subject Descriptors: