Abstract

Regularization plays a central role in the analysis of modern data, where non-regularized fitting is likely to lead to over-fitted models, useless for both prediction and interpretation. We consider the design of incremental algorithms which follow paths of regularized solutions, as the regularization varies. These approaches often result in methods which are both efficient and highly flexible. We suggest a general path-following algorithm based on second-order approximations, prove that under mild conditions it remains “very close ” to the path of optimal solutions and illustrate it with examples. 1

Citations

2529	C.J.E.: Uci repository of machine learning databases, http://www.ics.uci.edu/ ∼ mlearn/MLRepository.html - Blake, Keogh
998	Regression shrinkage and selection via the lasso - Tibshirani - 1996
107	The entire regularization path for the support vector machine - Hastie, Rosset, et al. - 2004
74	1-norm support vector machines - Zhu, Rosset, et al. - 2003
53	Piecewise linear regularized solution paths - Rosset, Zhu - 2003
51	Numerical Path Following - Allgower, Georg - 1997
49	Boosting as a regularized path to a maximum margin classifier - Rosset, Zhu, et al. - 2004
36	T: Classification of gene microarrays by penalized logistic regression - Zhu, Hastie
20	Soft Classification, a.k.a. Risk Estimation, via Penalized Log Likelihood and Smoothing Spline Analysis of Variance - Wahba, Gu, et al. - 1995
12	Gradient LASSO for feature selection - Kim, Kim - 2004
3	Elements of Stat - Hastie, Tibshirani, et al. - 2001
2	Topics in Regularization and Boosting - Rosset - 2003
1	Wavelet shrinkage: Asymptopia? Annals of Statistics - Donoho, Johnstone, et al. - 1995