DMCA
Efficient learning using forwardbackward splitting (2009)
| Venue: | in Advances in Neural Information Processing Systems 23 |
| Citations: | 24 - 0 self |
BibTeX
@INPROCEEDINGS{Duchi09efficientlearning,
author = {John Duchi and Yoram Singer},
title = {Efficient learning using forwardbackward splitting},
booktitle = {in Advances in Neural Information Processing Systems 23},
year = {2009}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
We describe, analyze, and experiment with a new framework for empirical loss minimization with regularization. Our algorithmic framework alternates between two phases. On each iteration we first perform an unconstrained gradient descent step. We then cast and solve an instantaneous optimization problem that trades off minimization of a regularization term while keeping close proximity to the result of the first phase. This yields a simple yet effective algorithm for both batch penalized risk minimization and online learning. Furthermore, the two phase approach enables sparse solutions when used in conjunction with regularization functions that promote sparsity, such as ℓ1. We derive concrete and very simple algorithms for minimization of loss functions with ℓ1, ℓ2, ℓ 2 2, and ℓ ∞ regularization. We also show how to construct efficient algorithms for mixed-norm ℓ1/ℓq regularization. We further extend the algorithms and give efficient implementations for very high-dimensional data with sparsity. We demonstrate the potential of the proposed framework in experiments with synthetic and natural datasets. 1
Keyphrases
forwardbackward splitting efficient implementation loss function unconstrained gradient descent step risk minimization high-dimensional data effective algorithm regularization term natural datasets efficient algorithm algorithmic framework alternate derive concrete sparse solution new framework phase approach empirical loss minimization online learning first phase close proximity simple algorithm regularization function instantaneous optimization problem
