Cookies on Macro by Mark.We use essential cookies for sign-in, locale, and security. With your permission, we also use optional cookies for preferences, for analytics, and for advertising measurement with the Meta (Facebook) Pixel. Read our cookie policy.
Sequential tree ensemble that fits residual structure stage by stage and uses validation loss to control overfitting.
Follow the same equations and assumptions one section at a time or in a continuous reading.
Hover or focus a symbol defined in this model to find its exact matches. Select it to keep the highlight; Escape clears the selection.
Gradient boosting reformulates the supervised learning problem as numerical optimization in function space. Instead of optimizing over a finite-dimensional parameter vector , the algorithm optimizes over the space of functions that minimize the expected loss . Friedman (2001) showed that steepest descent in this function space produces the boosting update rule: at each iteration, the algorithm computes the functional gradient (the direction of steepest descent) and projects it onto the space of base learners.
The key insight is that the negative gradient of the empirical loss with respect to the function values evaluated at the training points provides the direction of steepest descent. Since we cannot directly add an arbitrary function perturbation, we fit a base learner (tree) to approximate this gradient, then take a step in that direction. This is the bridge between abstract optimization in function space and the concrete algorithm of fitting trees to pseudo-residuals.
The empirical risk minimization objective. We seek the function that minimizes the total loss over the training data.
The pseudo-residual for observation at round . This is the negative gradient of the loss with respect to the current prediction, evaluated at . It points in the direction that would most reduce the loss for observation .
The base learner is fit to the pseudo-residuals by least squares. This projects the functional gradient onto the hypothesis class (e.g., regression trees with terminal nodes).
The boosting update. Add a shrunken version of the fitted tree to the current ensemble. The learning rate controls the step size in function space.
Why not use Newton's method directly?
Friedman's original GBM uses first-order gradient information only. This is analogous to gradient descent in parametric optimization: simple, stable, but potentially slow to converge. XGBoost's key innovation is to use second-order (Hessian) information, producing Newton-step updates that converge faster. The tradeoff: first-order methods are more robust to numerical issues in the Hessian (e.g., near-zero second derivatives), while second-order methods require the loss to be twice-differentiable and the Hessian to be positive.
Connection to boosting as forward stagewise additive modeling
Hastie, Tibshirani, and Friedman (2009, Ch. 10) showed that gradient boosting is a special case of forward stagewise additive modeling: at each stage, add the base learner that most reduces the objective, without revisiting earlier terms. When the loss is exponential and the base learners are classifiers, this recovers AdaBoost. When the loss is squared error and the base learners are trees, it recovers tree-based gradient boosting. The functional gradient framework unifies these special cases under a single optimization principle.
The plotted object solves an equilibrium system at each parameter vector. A path would require an additional law of motion, such as . Without that law, the model identifies endpoints but not the adjustment between them.
Read a parameter move as a new equilibrium or schedule position. The segment between two saved scenarios is not a transition law.
Do not infer speed, overshooting, or smooth adjustment from a static schedule shift.
A point or schedule is solved at the current parameter vector. It is not dated unless the model adds a transition law.
The parameter changes the solved object immediately in the diagram; the diagram does not say how institutions implement that change.
Proof lineage
Typed relationships across modelsParents -- models this proof draws from