Sepp Hochreiter and Jurgen Schmidhuber introduced the Long Short-Term Memory network in 1997 to solve a specific, well-characterized failure mode: the vanishing gradient problem in recurrent neural networks. Bengio, Simard, and Frasconi (1994) had already shown formally that gradients in vanilla RNNs shrink exponentially with sequence length during backpropagation through time (BPTT), making it impossible for the network to learn dependencies spanning more than roughly 10 to 20 time steps. Hochreiter's 1991 diploma thesis diagnosed the root cause as the repeated multiplication of the recurrent weight matrix, whose spectral radius determines whether gradients vanish (spectral radius below 1) or explode (spectral radius above 1). The LSTM solved this by introducing a dedicated cell state ct that flows through time with additive rather than multiplicative updates, regulated by multiplicative gates: the original 1997 design had input and output gates, and the forget gate that completes the modern three-gate cell was added by Gers, Schmidhuber, and Cummins (2000).
The core mechanism is the cell state highway. At each time step t, the cell state ct−1 from the previous step passes through a forget gate ft that element-wise scales it, then receives new candidate information c~t scaled by an input gate it: ct=ft⊙ct−1+it⊙c~t. The forget gate is a sigmoid-activated linear combination of the current input xt and previous hidden state ht−1: values near 1 preserve the cell content, values near 0 erase it. The input gate similarly controls how much of the new candidate (produced by a tanh-activated transform of xt and ht−1) is written into the cell. An output gate ot then selects which parts of the (tanh-squashed) cell state are exposed as the hidden state ht=ot⊙tanh(ct). This gating mechanism means the gradient can flow through the cell state with near-unity magnitude across many time steps, solving the vanishing gradient problem by construction.
LSTMs became the dominant sequence model in deep learning from roughly 2014 to 2018, spanning machine translation (Sutskever, Vinyals, and Le, 2014), speech recognition (Graves, Mohamed, and Hinton, 2013), and time-series forecasting. In macroeconomics, LSTMs have been deployed for GDP nowcasting, inflation forecasting, exchange rate prediction, and yield curve modeling. Nakamura (2005) was among the early applications to economic time series. More recently, the Federal Reserve Bank of New York and the Bank of Canada have experimented with LSTM-based nowcasting models that process high-frequency mixed-frequency inputs (daily financial indicators combined with monthly macro releases) in a sequential framework that naturally handles irregular arrival times. Makridakis, Spiliotis, and Assimakopoulos (2018) tested LSTMs in the M4 forecasting competition and found them competitive with statistical methods on longer series but less reliable on short ones, a pattern consistent with the data-hungry nature of neural sequence models.
The arrival of the Transformer architecture (Vaswani et al., 2017) reduced LSTM's dominance in NLP, but LSTMs remain competitive for univariate and low-dimensional multivariate time-series forecasting where the sequence is genuinely temporal and attention over thousands of positions is unnecessary. Salinas, Flunkert, Gasthaus, and Januschowski (2020) showed that LSTM-based DeepAR models produce well-calibrated probabilistic forecasts across thousands of related time series in Amazon's demand forecasting pipeline. For macro applications with moderate sequence lengths (60 to 240 monthly observations), LSTMs often outperform Transformers because the inductive bias of sequential processing matches the autoregressive structure of economic data, and the parameter count is lower, reducing overfitting risk on small samples.