Skip to main content

Cookies on Macro by Mark.We use essential cookies for sign-in, locale, and security. With your permission, we also use optional cookies for preferences, for analytics, and for advertising measurement with the Meta (Facebook) Pixel. Read our cookie policy.

Macro by MarkMacro by Mark
  • Home
Loading…
Loading…
← Models
MacroConceptsPolicyModels

Elastic Net Model

Regularized regression that blends LASSO selection with ridge shrinkage for correlated predictor blocks.

OverviewGraphMathReferences

Understand the mathematics

Follow the same equations and assumptions one section at a time or in a continuous reading.

Hover or focus a symbol defined in this model to find its exact matches. Select it to keep the highlight; Escape clears the selection.

Elastic Net Objective and Dual Penalty Geometry

The elastic net objective augments the OLS loss with a convex combination of L1 and L2 penalties. The resulting constraint set is the intersection of an L1 ball and an L2 ball, producing a shape with rounded corners that interpolates between the LASSO diamond and the ridge sphere.

Zou and Hastie (2005) noted that the naive elastic net (solving the penalized problem directly) suffers from double shrinkage: the L2 penalty shrinks coefficients, and then the L1 penalty shrinks them again. They proposed a rescaling correction: multiply the naive elastic net coefficients by (1+λ(1−α))(1 + \lambda(1 - \alpha))(1+λ(1−α)) to undo the extra L2 shrinkage. Modern implementations (glmnet) omit this rescaling and solve the naive elastic net directly - the coordinate descent update below is the naive one.

β^EN=arg⁡min⁡β12n∥y−Xβ∥22+λ[α∥β∥1+1−α2∥β∥22]\hat{\beta}_{\text{EN}} = \arg\min_\beta \frac{1}{2n} \|y - X\beta\|_2^2 + \lambda \left[ \alpha \|\beta\|_1 + \frac{1-\alpha}{2} \|\beta\|_2^2 \right]β^​EN​=argβmin​2n1​∥y−Xβ∥22​+λ[α∥β∥1​+21−α​∥β∥22​]
β^EN\hat{\beta}_{\text{EN}}β^​EN​nnnλ\lambdaλα\alphaα

The elastic net objective. The parameter α\alphaα controls the L1/L2 balance, and λ\lambdaλ controls overall penalty strength. At α=1\alpha = 1α=1 this is LASSO; at α=0\alpha = 0α=0 this is ridge.

Equivalently: min⁡β12n∥y−Xβ∥22+λ1∥β∥1+λ2∥β∥22,λ1=λα,  λ2=λ(1−α)/2\text{Equivalently: } \min_\beta \frac{1}{2n} \|y - X\beta\|_2^2 + \lambda_1 \|\beta\|_1 + \lambda_2 \|\beta\|_2^2, \quad \lambda_1 = \lambda\alpha, \; \lambda_2 = \lambda(1-\alpha)/2Equivalently: βmin​2n1​∥y−Xβ∥22​+λ1​∥β∥1​+λ2​∥β∥22​,λ1​=λα,λ2​=λ(1−α)/2
nnnλ\lambdaλα\alphaα

Two-parameter form. Some references use (λ1,λ2)(\lambda_1, \lambda_2)(λ1​,λ2​) instead of (λ,α)(\lambda, \alpha)(λ,α). The reparameterization is one-to-one for λ>0\lambda > 0λ>0.

Constraint form: min⁡β12n∥y−Xβ∥22s.t.α∥β∥1+1−α2∥β∥22≤t\text{Constraint form: } \min_\beta \frac{1}{2n}\|y - X\beta\|_2^2 \quad \text{s.t.} \quad \alpha\|\beta\|_1 + \frac{1-\alpha}{2}\|\beta\|_2^2 \leq tConstraint form: βmin​2n1​∥y−Xβ∥22​s.t.α∥β∥1​+21−α​∥β∥22​≤t
nnnα\alphaα

Lagrangian duality maps the penalty into a constraint. The feasible set is a blend of the L1 diamond and the L2 sphere. Its corners (where sparsity occurs) are smoothed by the L2 component.

β^ENcorrected=(1+λ(1−α))⋅β^ENnaive\hat{\beta}_{\text{EN}}^{\text{corrected}} = (1 + \lambda(1 - \alpha)) \cdot \hat{\beta}_{\text{EN}}^{\text{naive}}β^​ENcorrected​=(1+λ(1−α))⋅β^​ENnaive​
λ\lambdaλα\alphaα

Zou-Hastie rescaling. Corrects the double shrinkage of the naive elastic net. Without this, the L2 penalty over-shrinks coefficients that already survived L1 thresholding.

Write the penalized objective

Expand the elastic net penalty into its L1 and L2 components to prepare for coordinate-wise optimization.
Q(β)=12n∑i=1n(yi−xi′β)2+λα∑j=1p∣βj∣+λ(1−α)2∑j=1pβj2Q(\beta) = \frac{1}{2n} \sum_{i=1}^n (y_i - x_i'\beta)^2 + \lambda\alpha \sum_{j=1}^p |\beta_j| + \frac{\lambda(1-\alpha)}{2} \sum_{j=1}^p \beta_j^2Q(β)=2n1​i=1∑n​(yi​−xi′​β)2+λαj=1∑p​∣βj​∣+2λ(1−α)​j=1∑p​βj2​
nnnλ\lambdaλα\alphaα

Rewrite as augmented LASSO

Zou and Hastie's key insight: augmenting the data matrix converts elastic net into a LASSO problem on (X∗,y∗)(X^*, y^*)(X∗,y∗), so all LASSO theory applies to the augmented problem.
Define X∗=11+λ(1−α)(Xλ(1−α)Ip),y∗=(y0p)\text{Define } X^* = \frac{1}{\sqrt{1 + \lambda(1-\alpha)}} \begin{pmatrix} X \\ \sqrt{\lambda(1-\alpha)} I_p \end{pmatrix}, \quad y^* = \begin{pmatrix} y \\ 0_p \end{pmatrix}Define X∗=1+λ(1−α)​1​(Xλ(1−α)​Ip​​),y∗=(y0p​​)
λ\lambdaλα\alphaα

Verify equivalence

The augmented data adds ppp pseudo-observations that enforce L2 shrinkage through the identity block, while the L1 penalty on the augmented problem provides sparsity.
12(n+p)∥y∗−X∗β∗∥22+λα1+λ(1−α)∥β∗∥1⇔Q(β) up to rescaling\frac{1}{2(n+p)} \|y^* - X^* \beta^*\|_2^2 + \frac{\lambda\alpha}{\sqrt{1+\lambda(1-\alpha)}} \|\beta^*\|_1 \quad \Leftrightarrow \quad Q(\beta) \text{ up to rescaling}2(n+p)1​∥y∗−X∗β∗∥22​+1+λ(1−α)​λα​∥β∗∥1​⇔Q(β) up to rescaling
nnnλ\lambdaλα\alphaα

Constraint set geometry

In two dimensions, C(t)C(t)C(t) is a rounded diamond. As α→1\alpha \to 1α→1, the corners sharpen toward the LASSO diamond. As α→0\alpha \to 0α→0, the shape rounds into the ridge circle. OLS ellipses hitting a rounded corner set some coordinates to zero while keeping nearby coordinates nonzero.
C(t)={β:α∑j∣βj∣+1−α2∑jβj2≤t}C(t) = \left\{ \beta : \alpha \sum_j |\beta_j| + \frac{1-\alpha}{2} \sum_j \beta_j^2 \leq t \right\}C(t)={β:αj∑​∣βj​∣+21−α​j∑​βj2​≤t}
α\alphaα

Grouped selection bound

Zou and Hastie (2005), Theorem 1, which requires the two coefficients to agree in sign (β^jβ^k>0\hat{\beta}_j \hat{\beta}_k > 0β^​j​β^​k​>0). Under that hypothesis the coefficient difference is bounded by a function of 1−ρjk1 - \rho_{jk}1−ρjk​. When ρjk→1\rho_{jk} \to 1ρjk​→1, the bound goes to zero, forcing correlated predictors to have nearly identical coefficients.
If β^jENβ^kEN>0:∣β^jEN−β^kEN∣≤1λ(1−α)2(1−ρjk)⋅∥y∥1/n\text{If } \hat{\beta}_j^{\text{EN}} \hat{\beta}_k^{\text{EN}} > 0: \quad |\hat{\beta}_j^{\text{EN}} - \hat{\beta}_k^{\text{EN}}| \leq \frac{1}{\lambda(1-\alpha)} \sqrt{2(1 - \rho_{jk})} \cdot \|y\|_1 / nIf β^​jEN​β^​kEN​>0:∣β^​jEN​−β^​kEN​∣≤λ(1−α)1​2(1−ρjk​)​⋅∥y∥1​/n
λ\lambdaλα\alphaαnnn

Double shrinkage identification

The denominator applies L2 shrinkage on top of L1 soft-thresholding, producing coefficients that are biased toward zero more than LASSO. The rescaling correction multiplies by (1+λ(1−α))(1 + \lambda(1-\alpha))(1+λ(1−α)) to remove this extra layer.
β^jnaive=S(zj,αλ)1+λ(1−α)  ⟹  L2 factor 11+λ(1−α) shrinks beyond L1 threshold\hat{\beta}_j^{\text{naive}} = \frac{S(z_j, \alpha\lambda)}{1 + \lambda(1-\alpha)} \implies \text{L2 factor } \frac{1}{1+\lambda(1-\alpha)} \text{ shrinks beyond L1 threshold}β^​jnaive​=1+λ(1−α)S(zj​,αλ)​⟹L2 factor 1+λ(1−α)1​ shrinks beyond L1 threshold
α\alphaαλ\lambdaλ

Constraint set geometry in 2D

In two dimensions, the LASSO constraint is a diamond (rotated square), the ridge constraint is a circle, and the elastic net constraint is a rounded diamond whose curvature at the corners depends on α\alphaα. At α=0.5\alpha = 0.5α=0.5, the corners have noticeable rounding. This rounding is what prevents the OLS ellipse from landing exactly at a corner where one coordinate is zero while the other is large. Instead, the ellipse touches a smoothed face where both coordinates can be nonzero.

Augmented data trick

The augmented data construction is operational, not a proof device. It means any software that solves LASSO can solve elastic net by preprocessing the data. This is how early implementations worked before dedicated coordinate descent code existed. The augmentation adds p pseudo-observations, so the effective sample size is n + p, which is why elastic net can select more than n variables (the augmented problem has n + p rows).

Transition checks

Forecast origin
y^T+h∣T=ET[yT+h],h=1,2,…\widehat{y}_{T+h|T}=\mathbb{E}_T[y_{T+h}],\qquad h=1,2,\ldotsy​T+h∣T​=ET​[yT+h​],h=1,2,…

Conditioning on information at T separates the sample from the projection. The forecast can be smooth even when later realized data will not be.

The bridge point is shared so the eye can see the handoff. After that point, the red line is a model forecast, not observed history.

Do not score the forecast segment as realized data or read it as a policy promise.

Uncertainty propagation
Var⁡T(yT+h)=σ2∑j=0h−1ψj2\operatorname{Var}_T(y_{T+h})=\sigma^2\sum_{j=0}^{h-1}\psi_j^2VarT​(yT+h​)=σ2j=0∑h−1​ψj2​

Each future innovation contributes through its impulse weight ψj\psi_jψj​. Unless the future is degenerate, longer horizons carry more conditional variance.

The band widens when shocks accumulate through the model recursion.

Do not treat the band edge as an alternative deterministic scenario.

Variable timing

Observed history

yt∈ITy_t\in\mathcal{I}_Tyt​∈IT​

The history segment is realized data available at the forecast origin.

Conditional mean

y^T+h∣T=E[yT+h∣IT]\widehat{y}_{T+h|T}=\mathbb{E}[y_{T+h}\mid\mathcal{I}_T]y​T+h∣T​=E[yT+h​∣IT​]

The forecast path is a conditional expectation, so it can be smooth while future realizations remain volatile.

Prediction band

σh2=Var⁡(yT+h∣IT)\sigma_h^2=\operatorname{Var}(y_{T+h}\mid\mathcal{I}_T)σh2​=Var(yT+h​∣IT​)

Band width is horizon-specific uncertainty, not a second deterministic transition.

Proof lineage

Typed relationships across models

Parents -- models this proof draws from

ExtendsLASSOElastic net extends LASSO by adding an L2 penalty that stabilizes selection under predictor correlation and removes the min⁡(n,p)\min(n, p)min(n,p) variable cap.ExtendsRidge RegressionElastic net extends ridge by adding an L1 penalty that enables variable selection and sparsity, which ridge cannot produce.

Keep going

Read next

Continue with a related concept, model, or example.

Theory model

IS-LM Model

a well-connected theoretical model two steps from here

New angle: theoretical model →
Macro by Mark
Help
Privacy PolicyTerms of UseEthics and Compliance

© 2026 Mark Jayson Nation LLC