Abstract—Compressive sensing (CS) is an emerging field based on the revelation that a small collection of linear projections of a sparse signal contains enough information for stable, sub-Nyquist signal acquisition. When a statistical characterization of the signal is available, Bayesian inference can complement conventional CS methods based on linear programming or greedy algorithms. We perform asymptotically optimal Bayesian inference using belief propagation (BP) decoding, which represents the CS encoding matrix as a graphical model. Fast computation is obtained by reducing the size of the graphical model with sparse encoding matrices. To decode a length- signal containing large coefficients, our CS-BP decoding algorithm uses ( log ()) measurements and ( log 2 ()) computation. Finally, although we focus on a two-state mixture Gaussian model, CS-BP is easily adapted to other signal models. Index Terms—Bayesian inference, belief propagation, compressive sensing, fast algorithms, sparse matrices. I.

The goal of sparse approximation problems is to represent a target signal approximately as a linear combination of a few elementary signals drawn from a fixed collection. This paper surveys the major practical algorithms for sparse approximation. Specific attention is paid to computational issues, to the circumstances in which individual methods tend to perform well, and to the theoretical guarantees available. Many fundamental questions in electrical engineering, statistics, and applied mathematics can be posed as sparse approximation problems, making these algorithms versatile and relevant to a wealth of applications.

Abstract—Compressed sensing deals with the reconstruction of a high-dimensional signal from far fewer linear measurements, where the signal is known to admit a sparse representation in a certain linear space. The asymptotic scaling of the number of measurements needed for reconstruction as the dimension of the signal increases has been studied extensively. This work takes a fundamental perspective on the problem of inferring about individual elements of the sparse signal given the measurements, where the dimensions of the system become increasingly large. Using the replica method, the outcome of inferring about any fixed collection of signal elements is shown to be asymptotically decoupled, i.e., those elements become independent conditioned on the measurements. Furthermore, the problem of inferring about each signal element admits a single-letter characterization in the sense that the posterior distribution of the element, which is a sufficient statistic, becomes asymptotically identical to the posterior of inferring about the same element in scalar Gaussian noise. The result leads to simple characterization of all other elemental metrics of the compressed sensing problem, such as the mean squared error and the error probability for reconstructing the support set of the sparse signal. Finally, the single-letter characterization is rigorously justified in the special case of sparse measurement matrices where belief propagation becomes asymptotically optimal. I.

Abstract — This paper considers a simple on–off random multiple access channel, where n users communicate simultaneously to a single receiver over m degrees of freedom. Each user transmits with probability λ, where typically λn < m ≪ n, and the receiver must detect which users transmitted. We show that when the codebook has i.i.d. Gaussian entries, detecting which users transmitted is mathematically equivalent to a certain sparsity detection problem considered in compressed sensing. Using recent sparsity results, we derive upper and lower bounds on the capacities of these channels. We show that common sparsity detection algorithms, such as lasso and orthogonal matching pursuit (OMP), can be used as tractable multiuser detection schemes and have significantly better performance than single-user detection. These methods do achieve some near–far resistance but—at high signal-to-noise ratios (SNRs)—may achieve capacities far below optimal maximum likelihood detection. We then present a new algorithm, called sequential OMP, that illustrates that iterative detection combined with power ordering or power shaping can significantly improve the high SNR performance. Sequential OMP is analogous to successive interference cancellation in the classic multiple access channel. Our results thereby provide insight into the roles of power control and multiuser detection on random-access signalling. Index Terms — compressed sensing, convex optimization, lasso, maximum likelihood estimation, multiple access channel, multiuser detection, orthogonal matching pursuit, power control, random matrices, single-user detection, sparsity, thresholding I.

Abstract—Cleaning of noise from signals is a classical and longstudied problem in signal processing. Algorithms for this task necessarily rely on an a priori knowledge about the signal characteristics, along with information about the noise properties. For signals that admit sparse representations over a known dictionary, a commonly used denoising technique is to seek the sparsest representation that synthesizes a signal close enough to the corrupted one. As this problem is too complex in general, approximation methods, such as greedy pursuit algorithms, are often employed. In this line of reasoning, we are led to believe that detection of the sparsest representation is key in the success of the denoising goal. Does this mean that other competitive and slightly inferior sparse representations are meaningless? Suppose we are served with a group of competing sparse representations, each claiming to explain the signal differently. Can those be fused somehow to lead to a better result? Surprisingly, the answer to this question is positive; merging these representations can form a more accurate (in the mean-squared-error (MSE) sense), yet dense, estimate of the original signal even when the latter is known to be sparse. In this paper, we demonstrate this behavior, propose a practical way to generate such a collection of representations by randomizing the Orthogonal Matching Pursuit (OMP) algorithm, and produce a clear analytical justification for the superiority of the associated Randomized OMP (RandOMP) algorithm. We show that while the maximum a posteriori probability (MAP) estimator aims to find and use the sparsest representation, the minimum meansquared-error (MMSE) estimator leads to a fusion of representations to form its result. Thus, working with an appropriate mixture of candidate representations, we are surpassing the MAP and tending towards the MMSE estimate, and thereby getting a far more accurate estimation in terms of the expected `2-norm error. Index Terms—Bayesian, maximum a posteriori probability (MAP), minimum-mean-squared error (MMSE), sparse representations.

Abstract — A Bayesian approach is adopted for linear regression, and a fast algorithm is given for updating posterior probabilities. Emphasis is given to the underdetermined and sparse case, i.e., fewer observations than regression coefficients and the belief that only a few regression coefficients are non-zero. The fast update allows for a low-complexity method of reporting a set of models with high posterior probability and their exact posterior odds. As a byproduct, this Bayesian model averaged approach yields the minimum mean squared error estimate of unknown coefficients. Algorithm complexity is linear in the number of unknown coefficients, the number of observations and the number of nonzero coefficients. For the case in which hyperparameters are unknown, a maximum likelihood estimate is found by a generalized expectation maximization algorithm. I.

Channel Estimation is an essential component in applications such as radar and data communication. In multi path time varying environments, it is necessary to estimate time-shifts, scale-shifts (the wideband equivalent of Doppler-shifts), and the gains/phases of each of the multiple paths. With recent advances in sparse estimation (or “compressive sensing”), new estimation techniques have emerged which yield more accurate estimates of these channel parameters than traditional strategies. These estimation strategies, however, restrict potential estimates of time-shifts and scale-shifts to a finite set of values separated by a choice of grid spacing. A small grid spacing increases the number of potential estimates, thus lowering the quantization error, but also increases complexity and estimation time. Conversely, a large grid spacing lowers the number of potential estimates, thus lowering the complexity and estimation time, but increases the quantization error. In this thesis, we derive

Wyner-Ziv coding of continuous sources with uncertain side information quality is defined by modeling the correlation noise as a Gaussian-mixture. The analysis of the theoretical rate-distortion performance is presented, along with a coding solution not relying on the presence of a feedback channel. The attainable performance of the coding scheme is derived, and a brief discussion on implementation issues concludes the paper. 1.

Wyner-Ziv coding with uncertain side information quality is characterized introducing a Gaussian-mixture correlation model. The analysis of the theoretical rate-distortion performance is presented, along with a coding solution not relying on the presence of a feedback channel. Attainable performance of the coding scheme and implementation issues are discussed. I.