Skip to main content

Cookies on Macro by Mark.We use essential cookies for sign-in, locale, and security. With your permission, we also use optional cookies for preferences, for analytics, and for advertising measurement with the Meta (Facebook) Pixel. Read our cookie policy.

Macro by MarkMacro by Mark
  • Home
Loading…
Loading…
← Models
MacroConceptsPolicyModels

LASSO Model

L1-penalized linear regression - drives some coefficients to zero, doing variable selection inside the fit.

OverviewGraphMathReferences

Understand the mathematics

Follow the same equations and assumptions one section at a time or in a continuous reading.

Hover or focus a symbol defined in this model to find its exact matches. Select it to keep the highlight; Escape clears the selection.

Lasso Formulation and Soft Thresholding

The LASSO objective combines the OLS residual sum of squares with an L1 penalty on the coefficient vector. The L1 penalty is not differentiable at zero, which is precisely what enables exact zero-setting of coefficients.

The subgradient optimality condition yields the soft-thresholding rule for each coordinate. This is the foundation of coordinate descent, the workhorse algorithm for computing the LASSO path.

β^LASSO=arg⁡min⁡β12n∥y−Xβ∥22+λ∑j=1p∣βj∣\hat{\beta}_{\text{LASSO}} = \arg\min_\beta \frac{1}{2n}\|y - X\beta\|_2^2 + \lambda \sum_{j=1}^p |\beta_j|β^​LASSO​=argβmin​2n1​∥y−Xβ∥22​+λj=1∑p​∣βj​∣
β^LASSO\hat{\beta}_{\text{LASSO}}β^​LASSO​nnnλ\lambdaλ

The LASSO objective. The 1/(2n)1/(2n)1/(2n) normalization is conventional and ensures the gradient of the squared-error term is −n−1X′(y−Xβ)-n^{-1} X'(y - X\beta)−n−1X′(y−Xβ).

0∈−1nXj′(y−Xβ)+λ∂∣βj∣0 \in -\frac{1}{n} X_j'(y- X\beta) + \lambda \partial|\beta_j|0∈−n1​Xj′​(y−Xβ)+λ∂∣βj​∣
nnnλ\lambdaλ∂∣βj∣\partial |\beta_j|∂∣βj​∣

Subgradient optimality condition for coordinate jjj. The subdifferential of ∣βj∣|\beta_j|∣βj​∣ is {sign(βj)}\{\text{sign}(\beta_j)\}{sign(βj​)} when βj≠0\beta_j \neq 0βj​=0 and [−1,1][-1, 1][−1,1] when βj=0\beta_j = 0βj​=0.

β^j=S(Xj′(y−X−jβ^−j)Xj′Xj,  nλXj′Xj)\hat{\beta}_j = S\left( \frac{X_j'(y - X_{-j}\hat{\beta}_{-j})}{X_j' X_j}, \; \frac{n\lambda}{X_j' X_j} \right)β^​j​=S(Xj′​Xj​Xj′​(y−X−j​β^​−j​)​,Xj′​Xj​nλ​)
nnnλ\lambdaλ

Coordinate-wise update. With standardized predictors (Xj′Xj=nX_j'X_j = nXj′​Xj​=n), this simplifies to β^j=S(zj,λ)\hat{\beta}_j = S(z_j, \lambda)β^​j​=S(zj​,λ) where zj=Xj′rj/nz_j = X_j'r_j/nzj​=Xj′​rj​/n is the univariate regression coefficient of the partial residual on XjX_jXj​.

1

Isolate the contribution of predictor jjj by removing the fitted values from all other predictors.
Fix all coordinates except j. Define partial residual rj=y−∑k≠jXkβ^k\text{Fix all coordinates except } j. \text{ Define partial residual } r_j = y - \sum_{k \neq j} X_k \hat{\beta}_kFix all coordinates except j. Define partial residual rj​=y−k=j∑​Xk​β^​k​

2

With all other coordinates fixed, the problem reduces to a penalized univariate regression.
Univariate subproblem: min⁡βj12n∥rj−Xjβj∥2+λ∣βj∣\text{Univariate subproblem: } \min_{\beta_j} \frac{1}{2n}\|r_j - X_j \beta_j\|^2 + \lambda |\beta_j|Univariate subproblem: βj​min​2n1​∥rj​−Xj​βj​∥2+λ∣βj​∣
nnnλ\lambdaλ

3

The partial regression coefficient: the univariate OLS slope of rjr_jrj​ on XjX_jXj​.
zj=Xj′rjn(when Xj′Xj=n)z_j = \frac{X_j' r_j}{n} \quad (\text{when } X_j' X_j = n)zj​=nXj′​rj​​(when Xj′​Xj​=n)
zjz_jzj​nnn

4

This is the selection mechanism. The penalty kills coefficients whose univariate signal ∣zj∣|z_j|∣zj​∣ is below the threshold λ\lambdaλ.
If ∣zj∣≤λ, then β^j=0(the signal is too weak to survive the penalty)\text{If } |z_j| \leq \lambda, \text{ then } \hat{\beta}_j = 0 \quad (\text{the signal is too weak to survive the penalty})If ∣zj​∣≤λ, then β^​j​=0(the signal is too weak to survive the penalty)
zjz_jzj​λ\lambdaλ

5

Surviving coefficients are shrunk toward zero by exactly λ\lambdaλ. This is the shrinkage mechanism - selected variables have their estimated effects pulled toward zero.
If ∣zj∣>λ, then β^j=sign(zj)(∣zj∣−λ)\text{If } |z_j| > \lambda, \text{ then } \hat{\beta}_j = \text{sign}(z_j)(|z_j| - \lambda)If ∣zj​∣>λ, then β^​j​=sign(zj​)(∣zj​∣−λ)
zjz_jzj​λ\lambdaλ

6

Coordinate descent iterates until convergence. Each pass updates all ppp coordinates. Convergence is guaranteed because the objective is convex (strictly convex in the non-degenerate case).
Cycle j=1,…,p repeatedly until max⁡j∣Δβ^j∣<ε\text{Cycle } j = 1, \ldots, p \text{ repeatedly until } \max_j |\Delta \hat{\beta}_j| < \varepsilonCycle j=1,…,p repeatedly until jmax​∣Δβ^​j​∣<ε

L1 geometry

In two dimensions, the L1 constraint set {β:∣β1∣+∣β2∣≤t}\{\beta : |\beta_1| + |\beta_2| \leq t\}{β:∣β1​∣+∣β2​∣≤t} is a diamond (rotated square). The OLS contours (ellipses) are most likely to first touch this diamond at a corner, where one coordinate is zero. In ppp dimensions, the L1 ball has 2p2p2p corners and 2p2^p2p faces - the higher the dimension, the more likely the solution lands at a vertex or low-dimensional face, setting many coordinates to zero.

Notations

S(z,γ)=sign(z)(∣z∣−γ)+S(z, \gamma) = \text{sign}(z)(|z| - \gamma)_+S(z,γ)=sign(z)(∣z∣−γ)+​

Soft-thresholding operator

rj=y−X−jβ^−jr_j = y - X_{-j}\hat{\beta}_{-j}rj​=y−X−j​β^​−j​

Partial residual

zjz_jzj​

Univariate coefficient

∂∣βj∣\partial |\beta_j|∂∣βj​∣

Subdifferential

About these symbols

Soft-thresholding operator: soft-thresholding operator

Partial residual: partial residual for coordinate jjj

Univariate coefficient: univariate regression coefficient of rjr_jrj​ on XjX_jXj​

Subdifferential: subdifferential of the absolute value at βj\beta_jβj​

    Transition checks

    Forecast origin
    y^T+h∣T=ET[yT+h],h=1,2,…\widehat{y}_{T+h|T}=\mathbb{E}_T[y_{T+h}],\qquad h=1,2,\ldotsy​T+h∣T​=ET​[yT+h​],h=1,2,…

    Conditioning on information at T separates the sample from the projection. The forecast can be smooth even when later realized data will not be.

    The bridge point is shared so the eye can see the handoff. After that point, the red line is a model forecast, not observed history.

    Do not score the forecast segment as realized data or read it as a policy promise.

    Uncertainty propagation
    Var⁡T(yT+h)=σ2∑j=0h−1ψj2\operatorname{Var}_T(y_{T+h})=\sigma^2\sum_{j=0}^{h-1}\psi_j^2VarT​(yT+h​)=σ2j=0∑h−1​ψj2​

    Each future innovation contributes through its impulse weight ψj\psi_jψj​. Unless the future is degenerate, longer horizons carry more conditional variance.

    The band widens when shocks accumulate through the model recursion.

    Do not treat the band edge as an alternative deterministic scenario.

    Variable timing

    Observed history

    yt∈ITy_t\in\mathcal{I}_Tyt​∈IT​

    The history segment is realized data available at the forecast origin.

    Conditional mean

    y^T+h∣T=E[yT+h∣IT]\widehat{y}_{T+h|T}=\mathbb{E}[y_{T+h}\mid\mathcal{I}_T]y​T+h∣T​=E[yT+h​∣IT​]

    The forecast path is a conditional expectation, so it can be smooth while future realizations remain volatile.

    Prediction band

    σh2=Var⁡(yT+h∣IT)\sigma_h^2=\operatorname{Var}(y_{T+h}\mid\mathcal{I}_T)σh2​=Var(yT+h​∣IT​)

    Band width is horizon-specific uncertainty, not a second deterministic transition.

    Keep going

    Read next

    Continue with a related concept, model, or example.

    Theory model

    IS-LM Model

    a well-connected theoretical model two steps from here

    New angle: theoretical model →
    Macro by Mark
    Help
    Privacy PolicyTerms of UseEthics and Compliance

    © 2026 Mark Jayson Nation LLC