Skip to main content

Cookies on Macro by Mark.We use essential cookies for sign-in, locale, and security. With your permission, we also use optional cookies for preferences, for analytics, and for advertising measurement with the Meta (Facebook) Pixel. Read our cookie policy.

Macro by MarkMacro by Mark
  • Home
Loading…
Loading…
← Models
MacroConceptsPolicyModels

Gradient boosting Model

Sequential tree ensemble that fits residual structure stage by stage and uses validation loss to control overfitting.

OverviewGraphMathReferences

Understand the mathematics

Follow the same equations and assumptions one section at a time or in a continuous reading.

Hover or focus a symbol defined in this model to find its exact matches. Select it to keep the highlight; Escape clears the selection.

Functional Gradient Descent in Function Space

Gradient boosting reformulates the supervised learning problem as numerical optimization in function space. Instead of optimizing over a finite-dimensional parameter vector θ\thetaθ, the algorithm optimizes over the space of functions F:X→RF: \mathcal{X} \to \mathbb{R}F:X→R that minimize the expected loss L(F)=E[L(Y,F(X))]\mathcal{L}(F) = E[L(Y, F(X))]L(F)=E[L(Y,F(X))]. Friedman (2001) showed that steepest descent in this function space produces the boosting update rule: at each iteration, the algorithm computes the functional gradient (the direction of steepest descent) and projects it onto the space of base learners.

The key insight is that the negative gradient of the empirical loss with respect to the function values F(xi)F(x_i)F(xi​) evaluated at the training points provides the direction of steepest descent. Since we cannot directly add an arbitrary function perturbation, we fit a base learner (tree) to approximate this gradient, then take a step in that direction. This is the bridge between abstract optimization in function space and the concrete algorithm of fitting trees to pseudo-residuals.

F∗=arg⁡min⁡F∑i=1nL(yi,F(xi))F^* = \arg\min_{F} \sum_{i=1}^{n} L(y_i, F(x_i))F∗=argFmin​i=1∑n​L(yi​,F(xi​))

The empirical risk minimization objective. We seek the function FFF that minimizes the total loss over the training data.

rim=−[∂L(yi,F(xi))∂F(xi)]F=Fm−1r_{im} = -\left[\frac{\partial L(y_i, F(x_i))}{\partial F(x_i)}\right]_{F = F_{m-1}}rim​=−[∂F(xi​)∂L(yi​,F(xi​))​]F=Fm−1​​
rimr_{im}rim​

The pseudo-residual for observation iii at round mmm. This is the negative gradient of the loss with respect to the current prediction, evaluated at Fm−1(xi)F_{m-1}(x_i)Fm−1​(xi​). It points in the direction that would most reduce the loss for observation iii.

hm=arg⁡min⁡h∈H∑i=1n(rim−h(xi))2h_m = \arg\min_{h \in \mathcal{H}} \sum_{i=1}^{n} (r_{im} - h(x_i))^2hm​=argh∈Hmin​i=1∑n​(rim​−h(xi​))2
rimr_{im}rim​

The base learner hmh_mhm​ is fit to the pseudo-residuals by least squares. This projects the functional gradient onto the hypothesis class H\mathcal{H}H (e.g., regression trees with JJJ terminal nodes).

Fm(x)=Fm−1(x)+ν⋅hm(x),ν∈(0,1]F_m(x) = F_{m-1}(x) + \nu \cdot h_m(x), \quad \nu \in (0, 1]Fm​(x)=Fm−1​(x)+ν⋅hm​(x),ν∈(0,1]
Fm(x)F_m(x)Fm​(x)ν\nuνhm(x)h_m(x)hm​(x)

The boosting update. Add a shrunken version of the fitted tree to the current ensemble. The learning rate ν\nuν controls the step size in function space.

Initialize the ensemble

Start with a constant prediction that minimizes the loss. For squared error loss, F0=yˉF_0 = \bar{y}F0​=yˉ​. For log loss (binary classification), F0=log⁡(yˉ/(1−yˉ))F_0 = \log(\bar{y} / (1 - \bar{y}))F0​=log(yˉ​/(1−yˉ​)).
F0(x)=arg⁡min⁡γ∑i=1nL(yi,γ)F_0(x) = \arg\min_{\gamma} \sum_{i=1}^{n} L(y_i, \gamma)F0​(x)=argγmin​i=1∑n​L(yi​,γ)

Compute the functional gradient

The functional gradient at the nnn training points is an nnn-dimensional vector. Each component indicates how much the loss at observation iii would decrease per unit change in F(xi)F(x_i)F(xi​).
r~im=−∇F(xi)L(yi,Fm−1(xi))=−∂L(yi,F(xi))∂F(xi)∣F=Fm−1\tilde{r}_{im} = -\nabla_{F(x_i)} L(y_i, F_{m-1}(x_i)) = -\frac{\partial L(y_i, F(x_i))}{\partial F(x_i)} \bigg|_{F = F_{m-1}}r~im​=−∇F(xi​)​L(yi​,Fm−1​(xi​))=−∂F(xi​)∂L(yi​,F(xi​))​​F=Fm−1​​

Pseudo-residuals for common loss functions

For squared error loss L=12(y−F)2L = \frac{1}{2}(y - F)^2L=21​(y−F)2, the negative gradient is simply the residual. For absolute error L=∣y−F∣L = |y - F|L=∣y−F∣, it is the sign of the residual (median regression). For log loss in binary classification with σ\sigmaσ as the sigmoid, the negative gradient is the difference between the label and the predicted probability.
Squared error: rim=yi−Fm−1(xi).Absolute error: rim=sign(yi−Fm−1(xi)).Log loss: rim=yi−σ(Fm−1(xi))\text{Squared error: } r_{im} = y_i - F_{m-1}(x_i). \quad \text{Absolute error: } r_{im} = \text{sign}(y_i - F_{m-1}(x_i)). \quad \text{Log loss: } r_{im} = y_i - \sigma(F_{m-1}(x_i))Squared error: rim​=yi​−Fm−1​(xi​).Absolute error: rim​=sign(yi​−Fm−1​(xi​)).Log loss: rim​=yi​−σ(Fm−1​(xi​))
rimr_{im}rim​σ\sigmaσ

Project onto the base learner class

We cannot take an arbitrary step in function space because we are restricted to the hypothesis class H\mathcal{H}H (e.g., trees). Fitting a tree to the pseudo-residuals is the least-squares projection of the unrestricted gradient onto H\mathcal{H}H.
hm=arg⁡min⁡h∈H∑i=1n(rim−h(xi))2h_m = \arg\min_{h \in \mathcal{H}} \sum_{i=1}^{n} (r_{im} - h(x_i))^2hm​=argh∈Hmin​i=1∑n​(rim​−h(xi​))2
rimr_{im}rim​

Line search for optimal step size

After fixing the direction hmh_mhm​, a one-dimensional line search finds the step size ρm\rho_mρm​ that minimizes the loss along that direction. In practice, Friedman (2001) replaces the global line search with per-leaf optimization: each terminal node gets its own optimal constant.
ρm=arg⁡min⁡ρ∑i=1nL(yi,Fm−1(xi)+ρ⋅hm(xi))\rho_m = \arg\min_{\rho} \sum_{i=1}^{n} L(y_i, F_{m-1}(x_i) + \rho \cdot h_m(x_i))ρm​=argρmin​i=1∑n​L(yi​,Fm−1​(xi​)+ρ⋅hm​(xi​))

Per-leaf weight optimization

For a tree with JJJ terminal regions R1m,…,RJmR_{1m}, \ldots, R_{Jm}R1m​,…,RJm​, each leaf independently optimizes its weight wjmw_{jm}wjm​. This is more flexible than a single global step size because different regions of the feature space may need different magnitudes of correction.
wjm=arg⁡min⁡w∑i∈RjmL(yi,Fm−1(xi)+w),j=1,…,Jw_{jm} = \arg\min_{w} \sum_{i \in R_{jm}} L(y_i, F_{m-1}(x_i) + w), \quad j = 1, \ldots, Jwjm​=argwmin​i∈Rjm​∑​L(yi​,Fm−1​(xi​)+w),j=1,…,J
JJJ

Shrinkage regularization

The learning rate ν\nuν shrinks each tree's contribution below the line-search optimum. Friedman (2001) demonstrated empirically that ν<0.1\nu < 0.1ν<0.1 combined with many trees consistently outperforms ν=1\nu = 1ν=1 with fewer trees. The intuition is that small steps allow later trees to correct mistakes from multiple directions rather than committing fully to the first good direction.
Fm(x)=Fm−1(x)+ν∑j=1Jwjm⋅1(x∈Rjm),0<ν≤1F_m(x) = F_{m-1}(x) + \nu \sum_{j=1}^{J} w_{jm} \cdot \mathbf{1}(x \in R_{jm}), \quad 0 < \nu \leq 1Fm​(x)=Fm−1​(x)+νj=1∑J​wjm​⋅1(x∈Rjm​),0<ν≤1
Fm(x)F_m(x)Fm​(x)ν\nuνJJJ

Why not use Newton's method directly?

Friedman's original GBM uses first-order gradient information only. This is analogous to gradient descent in parametric optimization: simple, stable, but potentially slow to converge. XGBoost's key innovation is to use second-order (Hessian) information, producing Newton-step updates that converge faster. The tradeoff: first-order methods are more robust to numerical issues in the Hessian (e.g., near-zero second derivatives), while second-order methods require the loss to be twice-differentiable and the Hessian to be positive.

Connection to boosting as forward stagewise additive modeling

Hastie, Tibshirani, and Friedman (2009, Ch. 10) showed that gradient boosting is a special case of forward stagewise additive modeling: at each stage, add the base learner that most reduces the objective, without revisiting earlier terms. When the loss is exponential and the base learners are classifiers, this recovers AdaBoost. When the loss is squared error and the base learners are trees, it recovers tree-based gradient boosting. The functional gradient framework unifies these special cases under a single optimization principle.

Transition checks

Comparative-static map
E(θ)={x:F(x;θ)=0},θ0↦E(θ0), θ1↦E(θ1)E(\theta)=\{x: F(x;\theta)=0\},\qquad \theta_0\mapsto E(\theta_0),\ \theta_1\mapsto E(\theta_1)E(θ)={x:F(x;θ)=0},θ0​↦E(θ0​), θ1​↦E(θ1​)

The plotted object solves an equilibrium system at each parameter vector. A path would require an additional law of motion, such as xt+1=G(xt,θ)x_{t+1} = G(x_t, \theta)xt+1​=G(xt​,θ). Without that law, the model identifies endpoints but not the adjustment between them.

Read a parameter move as a new equilibrium or schedule position. The segment between two saved scenarios is not a transition law.

Do not infer speed, overshooting, or smooth adjustment from a static schedule shift.

Variable timing

Equilibrium object

F(x∗;θ)=0F(x^*;\theta)=0F(x∗;θ)=0

A point or schedule is solved at the current parameter vector. It is not dated unless the model adds a transition law.

Parameter move

θ0↦θ1\theta_0\mapsto\theta_1θ0​↦θ1​

The parameter changes the solved object immediately in the diagram; the diagram does not say how institutions implement that change.

Proof lineage

Typed relationships across models

Parents -- models this proof draws from

ExtendsRandom ForestGradient boosting and random forests are both tree ensembles, but with opposite construction strategies. Random forests build trees independently in parallel and average them (bagging). Gradient boosting builds trees sequentially, each correcting the errors of the prior ensemble. The sequential strategy typically achieves higher accuracy when tuned, while the parallel strategy is more robust to hyperparameter choices.ExtendsLASSOLASSO performs linear variable selection via L1 penalization. Gradient boosting extends beyond linearity by fitting nonlinear additive expansions of shallow trees, capturing interactions and thresholds that LASSO's linear constraint cannot represent.

Keep going

Read next

Continue with a related concept, model, or example.

Theory model

IS-LM Model

a well-connected theoretical model two steps from here

New angle: theoretical model →
Macro by Mark
Help
Privacy PolicyTerms of UseEthics and Compliance

© 2026 Mark Jayson Nation LLC