Citation Context

...bridge estimator can be viewed as the Bayes posterior mode under the prior pλ,q(β) = C(λ, q) exp(−λ|β|q). (8) Ridge regression (q = 2) corresponds to a Gaussian prior and the lasso (q = 1) a Laplacian (or double exponential) prior. The elastic net penalty corresponds to a new prior given by pλ,α(β) = C(λ, α) exp(−λ(α|β|2 + (1 − α)|β|1)); (9) a compromise between the Gaussian and Laplacian priors. Although bridge with 1 < q < 2 will have many similarities with the elastic net, there is a fundamental difference between them. The elastic nets produces sparse solutions, while the bridge does not. Fan & Li (2001) prove that in the Lq (q ≥ 1) penalty family, only the lasso penalty (q = 1) can produce a sparse solution. Bridge (1 < q < 2) always keeps all predictors in the model, as does ridge. Since automatic variable selection via penalization is a primary objective of this article, Lq (1 < q < 2) penalization is not a candidate. 3 Elastic Net 3.1 Deficiency of the naive elastic net As an automatic variable selection method, the naive elastic net overcomes the limitations of the lasso in scenarios (1) and (2). However, empirical evidence (see Sections 4 and 5) shows that the naive elastic net does not...

Citation Context

...part ℓ2.2 Regularization Paths for GLMs via Coordinate Descent 4. ℓ1 regularization paths for generalized linear models (Park and Hastie 2007a). 5. Methods using non-concave penalties, such as SCAD (=-=Fan and Li 2005-=-) and Friedman’s generalized elastic net (Friedman 2008), enforce more severe variable selection than the lasso. 6. Regularization paths for the support-vector machine (Hastie et al. 2004). 7. The gra...

Citation Context

...ntifies the right subset model, { j : β̂j "= 0} =A • Has the optimal estimation rate, √n(β̂(δ)A − β∗A) →d N(0,"∗), where "∗ is the covariance matrix knowing the true subset model. It has been argued (=-=Fan and Li 2001-=-; Fan and Peng 2004) that a good procedure should have these oracle properties. However, some extra conditions besides the oracle properties, such as continuous shrinkage, are also required in an opti...

Citation Context

...dures to select variables. Stepwise and subset selection procedures make these procedures computationally intensive, hard to derive sampling properties, and unstable (see for example Breiman 1996 and =-=Fan and Li 200-=-1). In contrast, most convex penalties, such as quadratic penalties, often produce shrinkage estimators of parameters that make trade-os between bias and variance such as those in smoothing splines. H...

Citation Context

...h in that special case it simplifies even further, or with a bridge penalty (1 < q < 2) proposed by Fu (1998), which may work better for certain classes of covariances. It can also be used with SCAD (=-=Fan and Li, 2001-=-) or other more complicated non-convex penalties that are typically approximated by the local quadratic approximation. Even though we derive the algorithm with a general q, in this paper we only prese...

Citation Context

...ut all of these are computationally intensive. A faster algorithm that employs the lasso was proposed by Friedman et al. [16]. This approach has also been extended to more general penalties like SCAD =-=[15]-=- by Lam and Fan [25] and Fan et al. [14]. In specific applications, there have been other permutation-invariant approaches that use different notions of sparsity: Zou et al. [35] apply the lasso penal...