#### DMCA

## Paved with good intentions: Analysis of a randomized Kaczmarz method (2012)

Citations: | 20 - 5 self |

### Citations

2704 |
Matrix Computations
- Golub, Loan
- 1996
(Show Context)
Citation Context ...r algebra are almost as old as the field itself. Gauss himself suggested a block version of his algorithm for solving linear systems [Gau95, Ben09]. Many other algorithms for numerical linear algebra =-=[GVL96]-=- and least-squares problems [Bjö96] admit natural block variants. The field of optimization also contains a wide variety of block methods, such as [AC89, Tse93]. We believe that row pavings can also p... |

860 |
Numerical Methods for Least Squares Problems
- Björck
- 1996
(Show Context)
Citation Context ...inverse A † τ to a vector. We can perform this calculation efficiently provided that each submatrix Aτ has well-conditioned rows. Indeed, in this case, we can invoke an iterative least-squares solver =-=[Bjö96]-=-, such as CGLS, to apply the pseudoinverse A † τ approximately using a small number of matrix–vector multiplies with Aτ and A ∗ τ . In particular, we never need to form the pseudoinverse. This observa... |

578 | Uncertainty principles and ideal atomic decomposition
- Donoho, Huo
(Show Context)
Citation Context ... an. We say that A is incoherent when max i=ℓ |〈ai cinc , aℓ〉| ≤ log(1 + n) . The number cinc is a positive, universal constant. Incoherent matrices arise, for example, in signal processing problems =-=[DH01]-=-. Every incoherent, standardized matrix admits a random paving with controlled lower and upper bounds. Proposition 3.5 (Random Paving). Suppose that A is an incoherent, standardized matrix with n rows... |

243 | Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions - Halko, Martinsson, et al. |

240 | User-friendly tail bounds for sums of random matrices - Tropp - 2010 |

225 |
Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography
- Gordon, Bender, et al.
- 1970
(Show Context)
Citation Context ...t opportunities for further research. 5.1. Kaczmarz Methods with Randomized Control. The Kaczmarz method was originally introduced in the paper [Kac37]. It was reinvented by researchers in tomography =-=[GBH70]-=- under the appellation “algebraic reconstruction technique” (ART). See Byrne’s book [Byr08] for a contemporary summary of this literature. The classical variants of the Kaczmarz method rely on determi... |

168 |
Angenäherte auflösung von systemen linearer gleichungen
- Kaczmarz
- 1937
(Show Context)
Citation Context ...e and construction of good row pavings. Together, these results yield an efficient block Kaczmarz scheme that applies to many overdetermined least-squares problem. 1. INTRODUCTION The Kaczmarz method =-=[Kac37]-=- is an iterative algorithm for solving overdetermined least-squares problems. Because of its simplicity and performance, this scheme has found application in fields ranging from image reconstruction t... |

108 | Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function
- Richtárik, Takác
- 2014
(Show Context)
Citation Context ...provide an analysis of a randomized iteration for solving least-squares problems with polyhedral constraints, while Richtárik and Takáč have extended these ideas to more general optimization problems =-=[RT11]-=-. Some other references include [EN11, RT11, CP12, NW12]. 5.2. Block Kaczmarz Methods. The block Kaczmarz update rule (1.5) we are studying is originally due to Elfving [Elf80, Eqn. (2.2)]. This updat... |

82 |
A bound on tail probabilities for quadratic forms in independent random variables
- Hanson, Wright
- 1971
(Show Context)
Citation Context ...bability.12 DEANNA NEEDELL AND JOEL A. TROPP We do not have a reference for Proposition 3.7, but the result is probably not new. Appendix A offers a short proof based on the Hanson–Wright inequality =-=[HW71]-=- for Rademacher chaos. Let us pause to examine the hypothesis (3.1). Recall that, for a standardized matrix A with n rows, the squared spectral norm ‖A‖ 2 attains its maximal value n when the rows are... |

75 | The method of alternating projections and the method of subspace corrections in Hilbert space
- Xu, Zikatanov
- 2002
(Show Context)
Citation Context ...is an extensive literature on the convergence behavior of block projection methods, a class of algorithms that includes block Kaczmarz schemes. In particular, we call out the work of Xu and Zikatanov =-=[XZ02]-=-, which contains a refined convergence analysis that applies to a wide range of algorithms. Nevertheless, to our knowledge, Algorithm 1.1 is the only block Kaczmarz method that offers an (expected) li... |

68 | The KadisonSinger problem in mathematics and engineering – a detailed account
- Casazza, Fickus, et al.
(Show Context)
Citation Context ...’s works [You12b, You12a] for the latest developments in this direction. Finally, let us mention that subset selection theorems have applications throughout mathematics and engineering. See the paper =-=[CT06]-=- for a discussion and references. 3.2. Randomized Methods for Paving. The second approach to paving has the benefit of utmost simplicity, but it is more limited in scope. The idea is to divide the row... |

67 |
Extensions of pure states
- Kadison, Singer
- 1959
(Show Context)
Citation Context ... to remove the logarithm. For general matrices, this question remains open. It is known [And79, BHKW88, BT91] that an affirmative answer would imply the long-standing conjecture of Kadison and Singer =-=[KS59]-=-. 3 A universal constant has no dependence on any parameter. 4 To verify this point, consider a large matrix A whose entries have equal magnitude and independent random signs. Use the Bai–Yin Law [BY9... |

67 | A randomized Kaczmarz algorithm with exponential convergence
- Strohmer, Vershynin
(Show Context)
Citation Context ... algorithm is easy to analyze, it is simple to implement, and it is often effective in practice. Our primary reference is the randomized Kaczmarz algorithm recently proposed by Strohmer and Vershynin =-=[SV09b]-=-. When A is standardized, their method operates as follows. At iteration j , independently of all previous random choices, the algorithm draws the row index t(j ) uniformly at random from the set {1,.... |

63 |
Algebraic reconstruction techniques can be made computationally efficient
- Herman, Meyer
- 1993
(Show Context)
Citation Context ...has long been known that the cyclic control scheme performs badly when the rows are arranged in an unhappy order [HS78].18 DEANNA NEEDELL AND JOEL A. TROPP The literature contains empirical evidence =-=[HM93]-=- that randomized control mechanisms may be more effective, but until recently there was no compelling theoretical analysis to support this observation. The paper [SV09b] of Strohmer and Vershynin is s... |

62 |
Invertibility of ’large’ submatrices with applications to the geometry of Banach spaces and harmonic analysis
- Bourgain, Tzafriri
- 1987
(Show Context)
Citation Context ...es O(nd) operations to achieve fixed precision in this setting. 3.2.3. Related Results. The idea that randomness might help us to construct a paving is already inherent in the Bourgain–Tzafriri paper =-=[BT87]-=- on the Restricted Invertibility Principle, where they use randomized row selection and matrix factorization to perform subset selection. A result [BT91, Thm. 2.3] in their subsequent paper demonstrat... |

61 | Limit of the smallest eigenvalue of a large-dimensional sample covariance matrix - Bai, Yin - 1993 |

60 | A fast randomized algorithm for the approximation of matrices
- Woolfe, Liberty, et al.
(Show Context)
Citation Context ...s of Ailon and Chazelle [AC09], who use the random matrix S to perform dimension reduction. We believe that the first application of S for randomized linear algebra appears in the paper Woolfe et al. =-=[WLRT08]-=-, where they use this transform to aid in computing matrix decompositions. See the works [AMT10, HMT11, BG12] for further results in this direction. Liberty’s dissertation [Lib09] describes other rand... |

56 | The fast Johnson–Lindenstrauss transform and approximate nearest neighbors
- Ailon, Chazelle
(Show Context)
Citation Context ...n and Tzafriri. On the other hand, the logarithmic factor is necessary at the incoherence level we have imposed [BT91, Ex. 2.2]. The fast incoherence transform is based on ideas of Ailon and Chazelle =-=[AC09]-=-, who use the random matrix S to perform dimension reduction. We believe that the first application of S for randomized linear algebra appears in the paper Woolfe et al. [WLRT08], where they use this ... |

54 | A Mathematical Introduction to Compressive Sensing - Foucart, Rauhut - 2013 |

54 | Supplementum theoriae combinationis observationum erroribus minimis obnoxiae. Commentatines societatis regiae scientarium Gottingensis recentiores - Gauss |

50 |
On a problem of Kadison and
- Bourgain, Tzafriri
- 1991
(Show Context)
Citation Context ...number css is a positive, universal constant. Proposition 3.1 follows from [Tro09, Thm. 1.2], once we track the parameter δ through the proof. This result has been attributed to Bourgain and Tzafriri =-=[BT91]-=-, but the earliest reference seems to be Vershynin’s paper [Ver06, Cor. 1.5]. See Section 3.1.1 for further background. Proposition 3.1 ensures that each standardized matrix contains a large set of we... |

47 | Block-iterative projection methods for parallel computation of solutions to convex feasibility problems. Linear Algebra and its Applications 120 - Aharoni, Censor - 1989 |

45 |
Iterative algorithms for large partitioned linear systems, with applications to image reconstruction
- Eggermont, Herman, et al.
- 1981
(Show Context)
Citation Context ...s of the Kaczmarz method suffice to reduce the error by a constant fraction, provided that the squared error is substantially larger than d‖e‖ 2 ∞ . 1.3. The Block Kaczmarz Method. In some situations =-=[EHL81]-=-, practitioners prefer to use a block version of the Kaczmarz method to solve the least-squares problem (1.1). We consider a formulation due to Elfving [Elf80]. This procedure begins with an initial g... |

41 | Improved analysis of the subsampled randomized hadamard transform
- Tropp
- 2011
(Show Context)
Citation Context ...obability at least 1 − n −1 . Proposition 3.3 results from an argument based on the matrix Chernoff inequality [Tro12, Thm. 1.1] and a union bound. A model for this type of proof appears in the paper =-=[Tro11]-=-. We omit the details. 5 A p × q matrix is tall when p ≥ q.PAVED WITH GOOD INTENTIONS 11 In contrast, if we wish to construct a paving with a nontrivial lower bound α, we must place additional assump... |

38 | Blendenpick: Supercharging LAPACK’s least-squares solvers - Avron, Maymounkov, et al. - 2010 |

38 |
Block-iterative methods for consistent and inconsistent linear equations
- Elfving
- 1980
(Show Context)
Citation Context ...econstruction to digital signal processing [SS87, CFM + 92, FS95, Nat01]. At each iteration, the basic Kaczmarz method makes progress by enforcing a single constraint, while the block Kaczmarz method =-=[Elf80]-=- enforces many constraints at once. This paper introduces a randomized version of the block Kaczmarz method that converges with an expected linear rate, and we characterize the performance of this alg... |

36 | Randomized methods for linear constraints: Convergence rates and conditioning
- Leventhal, Lewis
(Show Context)
Citation Context ... when the linear system is inconsistent [Nee10]. Zouzias and Freris [ZF12] exhibit a randomized procedure, based on ideas from [Pop98], that can reduce the size of the residual e. Leventhal and Lewis =-=[LL10]-=- provide an analysis of a randomized iteration for solving least-squares problems with polyhedral constraints, while Richtárik and Takáč have extended these ideas to more general optimization problems... |

30 | Randomized Kaczmarz solver for noisy linear systems
- Needell
(Show Context)
Citation Context ...[SV09b] provides a short, elegant proof that this iteration converges at an expected linear rate to the solution x⋆ of a consistent least-squares problem (i.e., where the residual e is zero). Needell =-=[Nee10]-=- has extended the argument of [SV09b] to the case of an inconsistent least-squares problem. For a standardized matrix A, Needell’s error estimate reads E‖x j − x⋆‖ 2 2 ≤ [ 1 − σ2 min (A) ] j ‖x0 − x⋆‖... |

28 | John’s decompositions: selecting a large part
- Vershynin
(Show Context)
Citation Context ...ese results are appealing because they construct the required subsets using an algorithmic procedure that admits a polynomial-time implementation. See [Sri10, Nao11] for further exposition. Vershynin =-=[Ver01]-=- has obtained a theory of subset selection for matrices that are not necessarily standardized. In his results, the squared Frobenius norm ‖A‖2 F plays the role of the number n of rows. Srivastava’s di... |

25 | Dual coordinate ascent methods for nonstrictly convex minimization - Tseng - 1993 |

22 | Extensions of block-projections methods with relaxation parameters to inconsistent and rank-deficient least-squares problems
- Popa
- 1998
(Show Context)
Citation Context ... Needell demonstrates that the randomized Kaczmarz method converges, even when the linear system is inconsistent [Nee10]. Zouzias and Freris [ZF12] exhibit a randomized procedure, based on ideas from =-=[Pop98]-=-, that can reduce the size of the residual e. Leventhal and Lewis [LL10] provide an analysis of a randomized iteration for solving least-squares problems with polyhedral constraints, while Richtárik a... |

21 |
Applied iterative methods
- Byrne
- 2008
(Show Context)
Citation Context ...czmarz method was originally introduced in the paper [Kac37]. It was reinvented by researchers in tomography [GBH70] under the appellation “algebraic reconstruction technique” (ART). See Byrne’s book =-=[Byr08]-=- for a contemporary summary of this literature. The classical variants of the Kaczmarz method rely on deterministic mechanisms for selecting a row at each iteration. Indeed, the simplest version just ... |

20 | Column subset selection, matrix factorization, and eigenvalue optimization
- Tropp
- 2009
(Show Context)
Citation Context ...ning rows {1,...,n}\τ1 to bite off another subset τ2, and so forth. After m ≤ Cpave · δ −2 ‖A‖ 2 log(1 + n) steps, we have exhausted the entire matrix. This argument yields Proposition 1.3. The paper =-=[Tro09]-=- contains an efficient computational method for identifying the subset τ promised by Proposition 3.1. This algorithm chooses a random set ω of rows from the matrix A with twice the cardinality of the ... |

19 | The mathematics of computerized tomography, volume 32 of Classics in Applied Mathematics - Natterer - 2001 |

19 | An elementary proof of the Restricted Invertibility Theorem
- Spielman, Srivastave
- 2012
(Show Context)
Citation Context ...ing result, Proposition 1.3, as a corollary of Proposition 3.1 also appears in the paper [BT91]. The last few years have witnessed some striking advances in this area. Indeed, Spielman and Srivastava =-=[SS12]-=- have recently invented an elementary proof of the Restricted Invertibility Principle. Their method only involves linear algebra, and it leads to sharp constants. Youssef [You12b, Thm. 4.2] has adapte... |

18 | Matrix norm inequalities and the relative Dixmier property, Integ. Eqns. and Operator Theory 11 - Berman, Halpern, et al. - 1988 |

18 | The random paving property for uniformly bounded matrices, Studia Mathematica 185
- Tropp
- 2008
(Show Context)
Citation Context ...under an incoherence assumption slightly stricter than Definition 3.4, they prove that a random partition induces a paving with const · ‖A‖ 4 blocks; no logarithmic factor is necessary. See the paper =-=[Tro08b]-=- for a modern proof of the latter result. The aforementioned theorems all yield the wrong dependence on the spectral norm in the size of a random row paving. The précis [Tro08a] shows how to obtain th... |

18 | Randomized extended Kaczmarz for solving least-squares
- Zouzias, Freris
(Show Context)
Citation Context ...s of the Kaczmarz scheme and related topics. In particular, Needell demonstrates that the randomized Kaczmarz method converges, even when the linear system is inconsistent [Nee10]. Zouzias and Freris =-=[ZF12]-=- exhibit a randomized procedure, based on ideas from [Pop98], that can reduce the size of the residual e. Leventhal and Lewis [LL10] provide an analysis of a randomized iteration for solving least-squ... |

16 | Improved matrix algorithms via the subsampled randomized hadamard transform - Boutsidis, Gittens - 2013 |

15 | Acceleration of randomized Kaczmarz method via the JohnsonLindenstrauss lemma - Eldar, Needell |

14 | Almost sure convergence of the Kaczmarz algorithm with random measurements - Chen, Powell - 2012 |

14 |
The angles between the null spaces of X-rays
- Hamaker, Solmon
- 1978
(Show Context)
Citation Context ...each iteration. Indeed, the simplest version just cycles through the rows in order. It has long been known that the cyclic control scheme performs badly when the rows are arranged in an unhappy order =-=[HS78]-=-.18 DEANNA NEEDELL AND JOEL A. TROPP The literature contains empirical evidence [HM93] that randomized control mechanisms may be more effective, but until recently there was no compelling theoretical... |

14 | Norms of random submatrices and sparse - Tropp - 2008 |

11 | New variants of the POCS method using affine subspaces of finite codimension, with applications to irregular sampling - Cenker, Feichtinger, et al. - 1992 |

11 | Sparse quadratic forms and their geometric applications (after batson, spielman and srivastava). arXiv preprint arXiv:1101.4324 - Naor - 2011 |

10 | Accelerated dense random projections
- Liberty
- 2009
(Show Context)
Citation Context ...paper Woolfe et al. [WLRT08], where they use this transform to aid in computing matrix decompositions. See the works [AMT10, HMT11, BG12] for further results in this direction. Liberty’s dissertation =-=[Lib09]-=- describes other randomized maps that can play a similar role.PAVED WITH GOOD INTENTIONS 13 4. NUMERICAL ASPECTS OF BLOCK KACZMARZ The main goal of this paper is to study the theoretical properties o... |

9 | A note on the behavior of the randomized Kaczmarz algorithm of Strohmer and Vershynin - Censor, Herman, et al. - 2009 |

9 | Two-subspace projection method for coherent overdetermined linear systems
- Needell, Ward
(Show Context)
Citation Context ...rz method. Figure 3 shows the results of this trial. We see that the simple Kaczmarz method scarcely reduces the error at all, while the block method achieves a healthy rate of convergence. The paper =-=[NW12]-=- provides an analysis of this example in the case where each block contains two rows, but we do not yet have a complete explanation for the performance of Algorithm 1.1 when the blocks are larger. Fig... |

9 | Incorporation of a priori moment information into signal recovery and synthesis problems - Sezan, Stark - 1987 |

8 | Block-projections algorithms with blocks containing mutually orthogonal rows and columns
- Popa
- 1999
(Show Context)
Citation Context ...ions we have in mind, this hypothesis is reasonable. Nevertheless, the block Kaczmarz method would be more versatile if we could construct row pavings for a broad class of matrices. To that end, Popa =-=[Pop99]-=- has developed an approach for producing a paving of a sparse matrix; see also [Pop01, Pop04]. But we can travel much farther down this road. It is an astonishing fact that every standardized matrix a... |

7 | Invertibility of random submatrices via tail decoupling and a matrix Chernoff inequality
- Chrétien, Darses
(Show Context)
Citation Context ...A whose paving bounds satisfy 1 − δ ≤ α ≤ β ≤ 1 + δ, with probability at least 1 − n −1 . Proposition 3.5 follows from [Tro08a, Cor. 5.2], along with some standard arguments [BT91, Tro08b]. The paper =-=[CD12]-=- contains superior estimates for the constants in this analysis. See Section 3.2.3 for some further background. Random paving is a striking idea because it is almost completely automatic. Given a guar... |

7 | A fast Kaczmarz-Kovarik algorithm for consistent least-squares problems - Popa |

7 | Comments on the randomized Kaczmarz method - Strohmer, Vershynin |

6 | Extensions, restrictions, and representations of states on C∗- algebras - Anderson - 1979 |

5 | Beneath the valley of the noncommutative arithmetic-geometric mean inequality: Conjectures, case studies, and consequences. Submitted for publication
- Recht, Re
- 2012
(Show Context)
Citation Context ... offer some numerical evidence that this alternative control scheme is more effective than the approach used in Algorithm 1.1. At present, compelling explanations for this phenomenon are lacking. See =-=[RR12]-=- for some discussion and conjectures. 4.3. Numerical Experiments. In this section, we present some numerical experiments to complement our discussions about the implementation and theoretical performa... |

5 | Random sets of isomorphism of linear operators on Hilbert space, High dimensional probability - Vershynin - 2006 |

4 |
A Kaczmarz-based approach to nonperiodic sampling on unions of rectangular lattices
- Feichtinger, Strohmer
- 1995
(Show Context)
Citation Context ...llect several batches of uniform time samples of a signal. Each sample set produces a set of equations that is easy to solve. The block Kaczmarz method provides an effective way to use this structure =-=[FS95]-=-. See Section 4.3.1 for a related numerical example. Second, the block Kaczmarz algorithm can be implemented more efficiently than the simple Kaczmarz algorithm in many computer architectures. This cl... |

4 | A Kaczmarz-Kovarik algorithm for symmetric ill-conditioned matrices - Popa |

4 | Restricted invertibility and the Banach–Mazur distance to the cube. Mathematika
- Youssef
- 2012
(Show Context)
Citation Context ...orm ‖A‖2F plays the role of the number n of rows. Srivastava’s dissertation [Sri10, Chap. 3] contains an algorithm for (weighted) subset selection that applies to general matrices. See Youssef’s work =-=[You12]-=- for the latest developments in this direction. Finally, let us mention that subset selection theorems have applications throughout mathematics and engineering. See the paper [CT06] for a discussion a... |

4 | Extensions, restrictions, and representations of states on C∗-algebras - Anderson - 1979 |

2 |
Some remarks on coordinate restriction of operators to coordinate subspaces
- Kashin, Tzafriri
- 1993
(Show Context)
Citation Context ...riri [BT87, Thm. 1.2], which guarantees that every standardized matrix contains a set of const/‖A‖2 rows whose minimal singular value is bounded away from zero. In a similar vein, Kashin and Tzafriri =-=[KT94]-=- establish that every standardized matrix contains a set of const/‖A‖2 rows whose norm is constant. Both results are based on random selection combined with matrix factorization. Neither result yields... |

2 | Spectral sparsification and restricted invertibility. Phd dissertation - Srivastava - 2010 |

1 | Key moments in the history of numerical analysis - Benzi - 2009 |

1 | A note on column subset selection. Available at arXiv:1212.0976 - Youssef - 2012 |