Jerome Friedman introduced the gradient boosting machine (GBM) in a 2001 paper that recast the problem of combining weak learners as numerical optimization in function space. Earlier boosting methods like AdaBoost (Freund and Schapire, 1997) operated through iterative reweighting of misclassified observations. Friedman showed that AdaBoost could be understood as steepest descent on an exponential loss, and generalized the idea to arbitrary differentiable loss functions. The core insight: at each iteration, compute the negative gradient of the loss with respect to the current ensemble prediction, fit a new base learner (typically a shallow regression tree) to those gradient values (pseudo-residuals), and add a shrunken version of the fitted tree to the ensemble. This sequential error-correction mechanism produces ensembles that are far more accurate than any individual tree, and the use of shallow trees (depth 3 to 8) means each base learner is a weak learner with high bias but low variance.
The mechanism operates by additive expansion in function space. At iteration m, the ensemble prediction is Fm(x)=Fm−1(x)+ν⋅hm(x), where hm is a regression tree fit to the pseudo-residuals rim=−∂L(yi,Fm−1(xi))/∂Fm−1(xi) and ν∈(0,1] is the learning rate (shrinkage). Each tree corrects the errors of the current ensemble rather than fitting the raw response. Shrinkage is critical: Friedman showed that reducing ν below 1.0 (typically 0.01 to 0.3) substantially improves generalization at the cost of requiring more trees. The combination of shallow trees and small learning rates means the ensemble builds up complexity gradually, capturing increasingly subtle patterns in the residual structure.
Chen and Guestrin (2016) introduced XGBoost, which extended Friedman's GBM with a second-order Taylor expansion of the loss function, explicit L2 regularization on leaf weights, and a penalty on tree complexity (number of leaves). The second-order expansion uses both the gradient gi and the Hessian hi of the loss, producing Newton-step updates that converge faster than the first-order gradient descent of classic GBM. XGBoost also introduced column subsampling (borrowed from random forests), histogram-based approximate splits, and system-level optimizations (cache-aware block structure, out-of-core computation) that made gradient boosting practical for datasets with millions of rows. LightGBM (Ke et al., 2017) and CatBoost (Prokhorenkova et al., 2018) followed with further algorithmic and engineering innovations, including leaf-wise growth and native categorical feature handling.
In macroeconomics, gradient boosting has become a primary tool for high-dimensional forecasting. Medeiros, Vasconcelos, Veiga, and Zilberman (2021) ran a large horse race for US CPI inflation on the FRED-MD panel: the headline winner was the random forest, with boosted trees among the nonlinear methods that beat penalized linear benchmarks. For Brazil, Garcia, Medeiros, and Vasconcelos (2017) documented the value of machine-learning methods for inflation forecasting in a data-rich environment. The Bank of England uses gradient boosting ensembles in its forecasting toolkit alongside neural networks and Bayesian VARs. Gu, Kelly, and Xiu (2020) showed gradient-boosted trees competitive with deep learning for cross-sectional asset pricing. The IMF uses XGBoost-based models in fiscal surveillance for revenue forecasting across heterogeneous country panels. Central banks in emerging economies have adopted gradient boosting for credit risk scoring and financial stability monitoring, where the sequential error-correction mechanism captures the nonlinear interaction between financial conditions and macroeconomic stress.