Skip to main content

Cookies on Macro by Mark.We use essential cookies for sign-in, locale, and security. With your permission, we also use optional cookies for preferences, for analytics, and for advertising measurement with the Meta (Facebook) Pixel. Read our cookie policy.

Macro by MarkMacro by Mark
  • Home
Loading…
Loading…
← Models
MacroConceptsPolicyModels

Ridge regression Model

L2-penalized linear regression - shrinks coefficients toward zero to stabilize forecasts when predictors are many or collinear.

OverviewGraphMathReferences

Understand the mathematics

Follow the same equations and assumptions one section at a time or in a continuous reading.

Hover or focus a symbol defined in this model to find its exact matches. Select it to keep the highlight; Escape clears the selection.

Ridge Estimator and Svd Representation

The ridge estimator minimizes the penalized residual sum of squares. The penalty adds λ\lambdaλ times the squared L2 norm of the coefficient vector to the OLS objective.

The SVD representation reveals the geometry: ridge regression shrinks each principal component of the regression toward zero by a factor that depends on the component's singular value relative to the penalty. Components with small singular values (those aligned with collinear directions) get shrunk most.

β^ridge=arg⁡min⁡β{∥y−Xβ∥22+λ∥β∥22}\hat{\beta}_{\text{ridge}} = \arg\min_\beta \left\{ \|y - X\beta\|_2^2 + \lambda \|\beta\|_2^2 \right\}β^​ridge​=argβmin​{∥y−Xβ∥22​+λ∥β∥22​}
β^ridge\hat{\beta}_{\text{ridge}}β^​ridge​λ\lambdaλ

The ridge objective. The first term is the usual sum of squared residuals. The second term penalizes large coefficients.

∂∂β[(y−Xβ)′(y−Xβ)+λβ′β]=−2X′y+2X′Xβ+2λβ=0\frac{\partial}{\partial\beta} \left[ (y - X\beta)'(y - X\beta) + \lambda \beta'\beta \right] = -2X'y + 2X'X\beta + 2\lambda\beta = 0∂β∂​[(y−Xβ)′(y−Xβ)+λβ′β]=−2X′y+2X′Xβ+2λβ=0
λ\lambdaλX′XX'XX′X

First-order condition. Setting the gradient to zero and solving for β\betaβ.

β^ridge=(X′X+λIp)−1X′y\hat{\beta}_{\text{ridge}} = (X'X + \lambda I_p)^{-1} X'yβ^​ridge​=(X′X+λIp​)−1X′y
β^ridge\hat{\beta}_{\text{ridge}}β^​ridge​X′XX'XX′Xλ\lambdaλ

Closed-form solution. The matrix X′X+λIpX'X + \lambda I_pX′X+λIp​ is always invertible for λ>0\lambda > 0λ>0, even when X′XX'XX′X is singular (p>np > np>n).

1

Singular value decomposition. djd_jdj​ are the singular values in decreasing order. The columns of VVV are the principal directions of the predictor space.
Write SVD: X=UDV′,U∈Rn×p,  D=diag(d1,…,dp),  V∈Rp×p\text{Write SVD: } X = U D V', \quad U \in \mathbb{R}^{n \times p}, \; D = \text{diag}(d_1, \ldots, d_p), \; V \in \mathbb{R}^{p \times p}Write SVD: X=UDV′,U∈Rn×p,D=diag(d1​,…,dp​),V∈Rp×p
X=UDV′X = U D V'X=UDV′nnnppp

2

Substituting the SVD. The inversion becomes diagonal in the VVV-rotated coordinate system.
X′X=VD2V′,(X′X+λI)−1=V(D2+λI)−1V′X'X = V D^2 V', \quad (X'X + \lambda I)^{-1} = V (D^2 + \lambda I)^{-1} V'X′X=VD2V′,(X′X+λI)−1=V(D2+λI)−1V′
X′XX'XX′Xλ\lambdaλ

3

SVD representation. Each principal component VjV_jVj​ of the prediction is shrunk by the factor dj2/(dj2+λ)d_j^2 / (d_j^2 + \lambda)dj2​/(dj2​+λ), which is close to 1 for large djd_jdj​ (well-determined directions) and close to 0 for small djd_jdj​ (collinear directions).
β^ridge=V(D2+λI)−1DU′y=Vdiag(djdj2+λ)U′y\hat{\beta}_{\text{ridge}} = V (D^2 + \lambda I)^{-1} D U' y = V \text{diag}\left( \frac{d_j}{d_j^2 + \lambda} \right) U' yβ^​ridge​=V(D2+λI)−1DU′y=Vdiag(dj2​+λdj​​)U′y
β^ridge\hat{\beta}_{\text{ridge}}β^​ridge​λ\lambdaλdjd_jdj​

4

The key quantity. When dj2≫λd_j^2 \gg \lambdadj2​≫λ, shrinkage is minimal (sj≈1s_j \approx 1sj​≈1). When dj2≪λd_j^2 \ll \lambdadj2​≪λ, shrinkage is near-total (sj≈0s_j \approx 0sj​≈0). This is why ridge handles collinearity: it zeroes out the poorly estimated directions.
Shrinkage factor for component j:sj(λ)=dj2dj2+λ\text{Shrinkage factor for component } j: \quad s_j(\lambda) = \frac{d_j^2}{d_j^2 + \lambda}Shrinkage factor for component j:sj​(λ)=dj2​+λdj2​​
λ\lambdaλ

5

The ridge hat matrix HλH_\lambdaHλ​ shrinks the fitted values in the same principal-component basis. Its trace gives the effective degrees of freedom.
y^ridge=Xβ^ridge=Udiag(sj)U′y=Hλy\hat{y}_{\text{ridge}} = X \hat{\beta}_{\text{ridge}} = U \text{diag}(s_j) U' y = H_\lambda yy^​ridge​=Xβ^​ridge​=Udiag(sj​)U′y=Hλ​y
β^ridge\hat{\beta}_{\text{ridge}}β^​ridge​HλH_\lambdaHλ​

6

Effective degrees of freedom. A continuous, monotonically decreasing function of λ\lambdaλ. At λ=0\lambda = 0λ=0, df=p\text{df} = pdf=p (OLS). As λ→∞\lambda \to \inftyλ→∞, df→0\text{df} \to 0df→0.
df(λ)=tr(Hλ)=∑j=1psj(λ)=∑j=1pdj2dj2+λ\text{df}(\lambda) = \text{tr}(H_\lambda) = \sum_{j=1}^p s_j(\lambda) = \sum_{j=1}^p \frac{d_j^2}{d_j^2 + \lambda}df(λ)=tr(Hλ​)=j=1∑p​sj​(λ)=j=1∑p​dj2​+λdj2​​
df(λ)\text{df}(\lambda)df(λ)HλH_\lambdaHλ​pppλ\lambdaλ

Bayesian interpretation

Ridge is the posterior mean of β\betaβ under the model y∣X,β∼N(Xβ,σ2I)y | X, \beta \sim N(X\beta, \sigma^2 I)y∣X,β∼N(Xβ,σ2I) with prior β∼N(0,(σ2/λ)I)\beta \sim N(0, (\sigma^2 / \lambda) I)β∼N(0,(σ2/λ)I). The penalty parameter maps to the prior variance: λ=σ2/τ2\lambda = \sigma^2 / \tau^2λ=σ2/τ2 where τ2\tau^2τ2 is the prior variance of each coefficient.

Notations

X=UDV′X = U D V'X=UDV′

Singular value decomposition

djd_jdj​

Singular value

sj(λ)=dj2/(dj2+λ)s_j(\lambda) = d_j^2 / (d_j^2 + \lambda)sj​(λ)=dj2​/(dj2​+λ)

Shrinkage factor

HλH_\lambdaHλ​

Ridge hat matrix

df(λ)\text{df}(\lambda)df(λ)

Effective degrees of freedom

About these symbols

Singular value decomposition: singular value decomposition

Singular value: jjj-th singular value of XXX

Shrinkage factor: shrinkage factor

Ridge hat matrix: ridge hat matrix

Effective degrees of freedom: effective degrees of freedom

    Transition checks

    Forecast origin
    y^T+h∣T=ET[yT+h],h=1,2,…\widehat{y}_{T+h|T}=\mathbb{E}_T[y_{T+h}],\qquad h=1,2,\ldotsy​T+h∣T​=ET​[yT+h​],h=1,2,…

    Conditioning on information at T separates the sample from the projection. The forecast can be smooth even when later realized data will not be.

    The bridge point is shared so the eye can see the handoff. After that point, the red line is a model forecast, not observed history.

    Do not score the forecast segment as realized data or read it as a policy promise.

    Uncertainty propagation
    Var⁡T(yT+h)=σ2∑j=0h−1ψj2\operatorname{Var}_T(y_{T+h})=\sigma^2\sum_{j=0}^{h-1}\psi_j^2VarT​(yT+h​)=σ2j=0∑h−1​ψj2​

    Each future innovation contributes through its impulse weight ψj\psi_jψj​. Unless the future is degenerate, longer horizons carry more conditional variance.

    The band widens when shocks accumulate through the model recursion.

    Do not treat the band edge as an alternative deterministic scenario.

    Variable timing

    Observed history

    yt∈ITy_t\in\mathcal{I}_Tyt​∈IT​

    The history segment is realized data available at the forecast origin.

    Conditional mean

    y^T+h∣T=E[yT+h∣IT]\widehat{y}_{T+h|T}=\mathbb{E}[y_{T+h}\mid\mathcal{I}_T]y​T+h∣T​=E[yT+h​∣IT​]

    The forecast path is a conditional expectation, so it can be smooth while future realizations remain volatile.

    Prediction band

    σh2=Var⁡(yT+h∣IT)\sigma_h^2=\operatorname{Var}(y_{T+h}\mid\mathcal{I}_T)σh2​=Var(yT+h​∣IT​)

    Band width is horizon-specific uncertainty, not a second deterministic transition.

    Keep going

    Read next

    Continue with a related concept, model, or example.

    Theory model

    IS-LM Model

    a well-connected theoretical model two steps from here

    New angle: theoretical model →
    Macro by Mark
    Help
    Privacy PolicyTerms of UseEthics and Compliance

    © 2026 Mark Jayson Nation LLC