Skip to main content

Cookies on Macro by Mark.We use essential cookies for sign-in, locale, and security. With your permission, we also use optional cookies for preferences, for analytics, and for advertising measurement with the Meta (Facebook) Pixel. Read our cookie policy.

Macro by MarkMacro by Mark
  • Home
Loading…
Loading…
← Models
MacroConceptsPolicyModels

Random forest Model

Bagged decision-tree ensemble for nonlinear macro prediction and feature-importance diagnostics.

OverviewGraphMathReferences

Understand the mathematics

Follow the same equations and assumptions one section at a time or in a continuous reading.

Hover or focus a symbol defined in this model to find its exact matches. Select it to keep the highlight; Escape clears the selection.

Variance Reduction from Bagging and Tree Decorrelation

The central statistical result behind random forests is that averaging BBB estimators reduces variance by a factor that depends on their pairwise correlation. When estimators are independent, averaging achieves the full 1/B1/B1/B variance reduction. When they are positively correlated (as bagged trees are), the reduction is limited by the average correlation. Random forests lower this correlation through feature subsampling.

Breiman (2001) formalized this using the decomposition of ensemble variance into a correlation component and an individual-tree variance component. The forest's prediction variance at a point xxx is Var[f^(x)]=ρˉ(x)σT2(x)+1−ρˉ(x)BσT2(x)\text{Var}[\hat{f}(x)] = \bar{\rho}(x) \sigma^2_T(x) + \frac{1 - \bar{\rho}(x)}{B} \sigma^2_T(x)Var[f^​(x)]=ρˉ​(x)σT2​(x)+B1−ρˉ​(x)​σT2​(x), where ρˉ(x)\bar{\rho}(x)ρˉ​(x) is the average pairwise correlation between tree predictions and σT2(x)\sigma^2_T(x)σT2​(x) is the variance of a single tree. The first term is irreducible by adding more trees; only reducing ρˉ\bar{\rho}ρˉ​ through feature subsampling can lower it.

f^(x)=1B∑b=1BTb(x)\hat{f}(x) = \frac{1}{B} \sum_{b=1}^{B} T_b(x)f^​(x)=B1​b=1∑B​Tb​(x)
f^(x)\hat{f}(x)f^​(x)BBBTb(x)T_b(x)Tb​(x)

The random forest prediction is the average of BBB individual tree predictions. For classification, this is a majority vote.

Var[f^(x)]=1B2∑b=1B∑b′=1BCov[Tb(x),Tb′(x)]\text{Var}[\hat{f}(x)] = \frac{1}{B^2} \sum_{b=1}^{B} \sum_{b'=1}^{B} \text{Cov}[T_b(x), T_{b'}(x)]Var[f^​(x)]=B21​b=1∑B​b′=1∑B​Cov[Tb​(x),Tb′​(x)]
f^(x)\hat{f}(x)f^​(x)BBBTb(x)T_b(x)Tb​(x)

Variance of the ensemble prediction expressed as the sum of all pairwise covariances between tree predictions.

=1B2[BσT2(x)+B(B−1)ρˉ(x)σT2(x)]=ρˉ(x)σT2(x)+1−ρˉ(x)BσT2(x)= \frac{1}{B^2} \left[ B \sigma^2_T(x) + B(B-1) \bar{\rho}(x) \sigma^2_T(x) \right] = \bar{\rho}(x) \sigma^2_T(x) + \frac{1 - \bar{\rho}(x)}{B} \sigma^2_T(x)=B21​[BσT2​(x)+B(B−1)ρˉ​(x)σT2​(x)]=ρˉ​(x)σT2​(x)+B1−ρˉ​(x)​σT2​(x)
BBB

Breiman's variance decomposition. The first term ρˉσT2\bar{\rho} \sigma^2_Tρˉ​σT2​ persists regardless of BBB. The second term vanishes as B→∞B \to \inftyB→∞. Reducing ρˉ\bar{\rho}ρˉ​ through feature subsampling directly lowers the irreducible first term.

lim⁡B→∞Var[f^(x)]=ρˉ(x)σT2(x)\lim_{B \to \infty} \text{Var}[\hat{f}(x)] = \bar{\rho}(x) \sigma^2_T(x)B→∞lim​Var[f^​(x)]=ρˉ​(x)σT2​(x)
f^(x)\hat{f}(x)f^​(x)

The asymptotic variance of an infinite forest. This is strictly positive whenever trees are correlated (ρˉ>0\bar{\rho} > 0ρˉ​>0), which is always the case for bagged trees. Reducing mtry reduces ρˉ\bar{\rho}ρˉ​ and pushes this floor lower.

Define tree variance and covariance

All trees are identically distributed (same bootstrap procedure and feature subsampling), so they share the same marginal variance. The average pairwise correlation ρˉ\bar{\rho}ρˉ​ captures how much information they share.
σT2(x)=Var[Tb(x)],Cov[Tb(x),Tb′(x)]=ρˉ(x)σT2(x) for b≠b′\sigma^2_T(x) = \text{Var}[T_b(x)], \quad \text{Cov}[T_b(x), T_{b'}(x)] = \bar{\rho}(x) \sigma^2_T(x) \text{ for } b \neq b'σT2​(x)=Var[Tb​(x)],Cov[Tb​(x),Tb′​(x)]=ρˉ​(x)σT2​(x) for b=b′
Tb(x)T_b(x)Tb​(x)

Expand ensemble variance

Separate the BBB diagonal terms (variance of each tree) from the B(B−1)B(B-1)B(B−1) off-diagonal terms (pairwise covariances). The exchangeability of trees lets us factor out σT2\sigma^2_TσT2​ and ρˉ\bar{\rho}ρˉ​.
Var[1B∑bTb]=1B2[∑bVar[Tb]+∑b≠b′Cov[Tb,Tb′]]=1B2[BσT2+B(B−1)ρˉσT2]\text{Var}\left[\frac{1}{B}\sum_b T_b\right] = \frac{1}{B^2}\left[\sum_b \text{Var}[T_b] + \sum_{b \neq b'} \text{Cov}[T_b, T_{b'}]\right] = \frac{1}{B^2}\left[B\sigma^2_T + B(B-1)\bar{\rho}\sigma^2_T\right]Var[B1​b∑​Tb​]=B21​​b∑​Var[Tb​]+b=b′∑​Cov[Tb​,Tb′​]​=B21​[BσT2​+B(B−1)ρˉ​σT2​]
BBB

Simplify to Breiman decomposition

Collect terms. The first piece ρˉσT2\bar{\rho}\sigma^2_Tρˉ​σT2​ does not decrease with BBB. The second piece decreases as 1/B1/B1/B and vanishes for large forests.
=σT2B+B−1BρˉσT2=ρˉσT2+1−ρˉBσT2= \frac{\sigma^2_T}{B} + \frac{B-1}{B}\bar{\rho}\sigma^2_T = \bar{\rho}\sigma^2_T + \frac{1 - \bar{\rho}}{B}\sigma^2_T=BσT2​​+BB−1​ρˉ​σT2​=ρˉ​σT2​+B1−ρˉ​​σT2​
BBB

Effect of mtry on correlation

When mmm is small, each tree sees a different feature subset at each split, producing different partitions even from the same bootstrap sample. When m=pm = pm=p, trees differ only through bootstrap variation, and the dominant predictor appears at the root of every tree.
ρˉ(m) is increasing in m.ρˉ(1)≈0  (random splits),ρˉ(p)=ρˉbag  (full bagging)\bar{\rho}(m) \text{ is increasing in } m. \quad \bar{\rho}(1) \approx 0 \;\text{(random splits)}, \quad \bar{\rho}(p) = \bar{\rho}_{\text{bag}} \;\text{(full bagging)}ρˉ​(m) is increasing in m.ρˉ​(1)≈0(random splits),ρˉ​(p)=ρˉ​bag​(full bagging)
mmm

Bias-variance tradeoff in mtry

Decreasing mtry reduces ρˉ\bar{\rho}ρˉ​ (lower variance) but increases bias because individual trees are weaker (they miss relevant predictors at some splits). The optimal mtry minimizes this tradeoff.
MSE[f^]=Bias2[f^]+Var[f^]=Bias2[f^]+ρˉ(m)σT2(m)+1−ρˉ(m)BσT2(m)\text{MSE}[\hat{f}] = \text{Bias}^2[\hat{f}] + \text{Var}[\hat{f}] = \text{Bias}^2[\hat{f}] + \bar{\rho}(m)\sigma^2_T(m) + \frac{1 - \bar{\rho}(m)}{B}\sigma^2_T(m)MSE[f^​]=Bias2[f^​]+Var[f^​]=Bias2[f^​]+ρˉ​(m)σT2​(m)+B1−ρˉ​(m)​σT2​(m)
mmmBBB

Comparison with plain bagging

Since ρˉRF<ρˉbag\bar{\rho}_{\text{RF}} < \bar{\rho}_{\text{bag}}ρˉ​RF​<ρˉ​bag​ (feature subsampling decorrelates trees), the random forest has strictly lower variance than plain bagging for any finite or infinite BBB.
VarRF=ρˉRFσT2+1−ρˉRFBσT2<ρˉbagσT2+1−ρˉbagBσT2=Varbag\text{Var}_{\text{RF}} = \bar{\rho}_{\text{RF}} \sigma^2_T + \frac{1 - \bar{\rho}_{\text{RF}}}{B} \sigma^2_T < \bar{\rho}_{\text{bag}} \sigma^2_T + \frac{1 - \bar{\rho}_{\text{bag}}}{B} \sigma^2_T = \text{Var}_{\text{bag}}VarRF​=ρˉ​RF​σT2​+B1−ρˉ​RF​​σT2​<ρˉ​bag​σT2​+B1−ρˉ​bag​​σT2​=Varbag​
BBB

Why not set mtry = 1?

At m=1m = 1m=1, each split is chosen from a single randomly selected feature. Trees become nearly independent (ρˉ≈0\bar{\rho} \approx 0ρˉ​≈0) and ensemble variance approaches σT2/B\sigma^2_T / BσT2​/B, which is excellent. But individual trees are so poor that bias dominates. A tree that can only split on random single features is essentially a random partition of the space. The ensemble of random partitions converges to something useful but requires thousands of trees and large samples. The default m=p/3m = p/3m=p/3 provides a practical sweet spot.

Finite-B practice

In practice, the second term 1−ρˉBσT2\frac{1 - \bar{\rho}}{B} \sigma^2_TB1−ρˉ​​σT2​ becomes negligible by B=200B = 200B=200 to 500500500 for most macro applications. The OOB error curve flattens visibly at this point. Growing more trees never hurts prediction accuracy (the forest prediction is a running average), but it does increase computation and memory. The convergence diagnostic is the OOB error curve itself: stop when the curve is flat.

Transition checks

Comparative-static map
E(θ)={x:F(x;θ)=0},θ0↦E(θ0), θ1↦E(θ1)E(\theta)=\{x: F(x;\theta)=0\},\qquad \theta_0\mapsto E(\theta_0),\ \theta_1\mapsto E(\theta_1)E(θ)={x:F(x;θ)=0},θ0​↦E(θ0​), θ1​↦E(θ1​)

The plotted object solves an equilibrium system at each parameter vector. A path would require an additional law of motion, such as xt+1=G(xt,θ)x_{t+1} = G(x_t, \theta)xt+1​=G(xt​,θ). Without that law, the model identifies endpoints but not the adjustment between them.

Read a parameter move as a new equilibrium or schedule position. The segment between two saved scenarios is not a transition law.

Do not infer speed, overshooting, or smooth adjustment from a static schedule shift.

Variable timing

Equilibrium object

F(x∗;θ)=0F(x^*;\theta)=0F(x∗;θ)=0

A point or schedule is solved at the current parameter vector. It is not dated unless the model adds a transition law.

Parameter move

θ0↦θ1\theta_0\mapsto\theta_1θ0​↦θ1​

The parameter changes the solved object immediately in the diagram; the diagram does not say how institutions implement that change.

Proof lineage

Typed relationships across models

Parents -- models this proof draws from

ExtendsElastic NetRandom forests extend beyond penalized linear methods by capturing nonlinear interactions and threshold effects without specifying a functional form. Elastic net handles correlated linear predictors; random forests handle nonlinear relationships among those same predictors.ExtendsLASSOLASSO performs linear variable selection via L1 penalization. Random forests perform implicit nonlinear variable selection via tree splits and importance measures, without the linearity assumption.

Keep going

Read next

Continue with a related concept, model, or example.

Theory model

IS-LM Model

a well-connected theoretical model two steps from here

New angle: theoretical model →
Macro by Mark
Help
Privacy PolicyTerms of UseEthics and Compliance

© 2026 Mark Jayson Nation LLC