Jerome Friedman introduced the gradient boosting machine (GBM) in a 2001 paper that recast the problem of combining weak learners as numerical optimization in function space. Earlier boosting methods like AdaBoost (Freund and Schapire, 1997) operated through iterative reweighting of misclassified observations. Friedman showed that AdaBoost could be understood as steepest descent on an exponential loss, and generalized the idea to arbitrary differentiable loss functions. The core insight: at each iteration, compute the negative gradient of the loss with respect to the current ensemble prediction, fit a new base learner (typically a shallow regression tree) to those gradient values (pseudo-residuals), and add a shrunken version of the fitted tree to the ensemble. This sequential error-correction mechanism produces ensembles that are far more accurate than any individual tree, and the use of shallow trees (depth 3 to 8) means each base learner is a weak learner with high bias but low variance.

The mechanism operates by additive expansion in function space. At iteration $m$, the ensemble prediction is $F_m(x) = F_{m-1}(x) + \nu \cdot h_m(x)$, where $h_m$ is a regression tree fit to the pseudo-residuals $r_{im} = -\partial L(y_i, F_{m-1}(x_i)) / \partial F_{m-1}(x_i)$ and $\nu \in (0, 1]$ is the learning rate (shrinkage). Each tree corrects the errors of the current ensemble rather than fitting the raw response. Shrinkage is critical: Friedman showed that reducing $\nu$ below 1.0 (typically 0.01 to 0.3) substantially improves generalization at the cost of requiring more trees. The combination of shallow trees and small learning rates means the ensemble builds up complexity gradually, capturing increasingly subtle patterns in the residual structure.

Chen and Guestrin (2016) introduced XGBoost, which extended Friedman's GBM with a second-order Taylor expansion of the loss function, explicit L2 regularization on leaf weights, and a penalty on tree complexity (number of leaves). The second-order expansion uses both the gradient $g_i$ and the Hessian $h_i$ of the loss, producing Newton-step updates that converge faster than the first-order gradient descent of classic GBM. XGBoost also introduced column subsampling (borrowed from random forests), histogram-based approximate splits, and system-level optimizations (cache-aware block structure, out-of-core computation) that made gradient boosting practical for datasets with millions of rows. LightGBM (Ke et al., 2017) and CatBoost (Prokhorenkova et al., 2018) followed with further algorithmic and engineering innovations, including leaf-wise growth and native categorical feature handling.

In macroeconomics, gradient boosting has become a primary tool for high-dimensional forecasting. Medeiros, Vasconcelos, Veiga, and Zilberman (2021) found that boosted trees outperformed random forests and penalized linear models for Brazilian inflation forecasting with over 100 predictors. The Bank of England uses gradient boosting ensembles in its forecasting toolkit alongside neural networks and Bayesian VARs. Gu, Kelly, and Xiu (2020) showed gradient-boosted trees competitive with deep learning for cross-sectional asset pricing. The IMF uses XGBoost-based models in fiscal surveillance for revenue forecasting across heterogeneous country panels. Central banks in emerging economies have adopted gradient boosting for credit risk scoring and financial stability monitoring, where the sequential error-correction mechanism captures the nonlinear interaction between financial conditions and macroeconomic stress.

Inputs follow the standard supervised learning setup: $n$ observations of a target $y$ (GDP growth, inflation, recession indicator) and $p$ predictor variables $X$ (financial indices, labor market indicators, commodity prices, survey expectations, and their lags). Unlike random forests, gradient boosting is sensitive to the scale of the loss function, so practitioners typically difference nonstationary series and may standardize predictors when using custom loss functions. The algorithm handles mixed types natively through tree splits. Missing values are handled by default direction assignments at each split node: during training, the algorithm learns whether observations with missing values for a given feature should go left or right, based on which direction minimizes the loss.

The model is an additive ensemble of $M$ regression trees: $F_M(x) = F_0 + \sum_{m=1}^{M} \nu \cdot h_m(x)$, where $F_0$ is an initial constant (typically the mean of $y$ for squared error loss), $h_m$ is a shallow tree with $J$ terminal nodes, and $\nu$ is the learning rate. Each tree $h_m$ is fit not to the response $y$ but to the pseudo-residuals $r_{im} = -\left[\partial L(y_i, F(x_i)) / \partial F(x_i)\right]_{F=F_{m-1}}$. For squared error loss, the pseudo-residuals are simply the ordinary residuals $y_i - F_{m-1}(x_i)$. For other losses (absolute error, quantile, Huber), the pseudo-residuals take different forms, allowing gradient boosting to optimize any differentiable objective. Tree depth $J$ controls the interaction order: a tree with $J$ leaves can capture at most $(J-1)$-way interactions. Typical values are $J = 4$ to $8$ for macro applications, allowing 3- to 7-way interactions.

Four hyperparameters dominate the bias-variance tradeoff. The learning rate $\nu$ controls how much each tree contributes: smaller values require more trees but produce smoother generalization curves. The number of boosting rounds $M$ determines total model complexity; too many rounds lead to overfitting. Tree depth $J$ sets the interaction order. The subsampling fraction $\eta$ (row subsampling per round, typically 0.5 to 0.8) introduces stochastic gradient boosting (Friedman, 2002), which acts as additional regularization and speeds up training. XGBoost adds further controls: L2 regularization on leaf weights $\lambda$, L1 regularization $\alpha$, minimum child weight (sum of Hessians in a leaf), and column subsampling per tree or per split. Early stopping on a validation set is the standard method for selecting $M$: training continues until the validation loss has not improved for a specified patience window.

Medeiros, Vasconcelos, Veiga, and Zilberman (2021) conducted a comprehensive forecasting comparison for Brazilian macroeconomic variables using 117 predictors. Gradient-boosted trees with early stopping consistently outperformed random forests, LASSO, and ridge regression for inflation and industrial production at horizons of 1 to 12 months. The key advantage was the sequential error correction: after the first 50 to 100 rounds captured the dominant linear signals (comparable to what ridge or LASSO extracts), additional rounds picked up nonlinear interaction patterns in the residuals that flat ensemble methods like random forests spread across all trees simultaneously. The Bank of England's forecasting toolkit includes XGBoost models for GDP nowcasting, where the algorithm's ability to optimize asymmetric loss functions (quantile loss for downside risk) provides density forecasts that penalized linear models cannot match without distributional assumptions.

In financial economics, Gu, Kelly, and Xiu (2020) tested gradient-boosted trees against neural networks, random forests, and penalized linear models for monthly stock return prediction using 94 firm-level characteristics. Boosted trees ranked among the top methods, with the largest gains over linear models coming from their ability to capture conditional interactions between valuation ratios and momentum signals. The Federal Reserve Bank of Chicago uses gradient boosting in its National Financial Conditions Index construction, where the sequential fitting process naturally downweights redundant financial indicators. Credit risk applications at the ECB and BIS use XGBoost for probability-of-default modeling, where the built-in handling of missing data (common in credit bureau records) and the Hessian-weighted splits (which account for the curvature of the log-likelihood in logistic regression) produce well-calibrated probability estimates.

Gradient boosting should not be used when the analyst lacks a proper validation framework for hyperparameter tuning. Unlike random forests, which perform reasonably with default settings, gradient boosting is sensitive to the joint configuration of learning rate, tree depth, subsampling fraction, and regularization strength. A poorly tuned boosted model can badly overfit or underfit. The method also shares the extrapolation limitation of all tree-based approaches: predictions for feature values outside the training range revert to boundary leaf values. For macro stress testing or scenario analysis involving unprecedented conditions, penalized linear models or Bayesian VARs that can extrapolate trends are preferable. When structural coefficients with standard errors are needed (the slope of the Phillips curve, the fiscal multiplier), gradient boosting provides importance rankings and partial dependence plots but not regression coefficients. See the random forest reference at /models/empirical/random-forest for the parallel ensemble alternative that trades peak accuracy for tuning robustness.

Causal inference extensions of gradient boosting have emerged through the work on causal forests (Athey and Imbens, 2016) and the Bayesian causal forest framework (Hahn, Murray, Carvalho, 2020). While these are more commonly implemented via random forest variants, the gradient boosting sequential-correction principle has been adapted for heterogeneous treatment effect estimation in policy evaluation. Fiscal multiplier heterogeneity across business cycle states, monetary policy transmission differences across credit conditions, and targeted intervention effects in development economics all benefit from the boosting framework's ability to concentrate model capacity on the hardest-to-predict subpopulations.