Skip to main content

Cookies on Macro by Mark.We use essential cookies for sign-in, locale, and security. With your permission, we also use optional cookies for preferences, for analytics, and for advertising measurement with the Meta (Facebook) Pixel. Read our cookie policy.

Macro by MarkMacro by Mark
  • Home
Loading…
Loading…
← Models
MacroConceptsPolicyModels

LSTM Model

Recurrent neural network architecture with gates that can retain or forget information across long macro sequences.

OverviewGraphMathReferences

Understand the mathematics

Follow the same equations and assumptions one section at a time or in a continuous reading.

Hover or focus a symbol defined in this model to find its exact matches. Select it to keep the highlight; Escape clears the selection.

Lstm Forward Pass: Gate Equations and Cell State Update

The LSTM forward pass at each time step ttt computes four quantities in parallel from the concatenation of the previous hidden state ht−1h_{t-1}ht−1​ and the current input xtx_txt​: the forget gate ftf_tft​, the input gate iti_tit​, the candidate cell state c~t\tilde{c}_tc~t​, and the output gate oto_tot​. These gating values then combine to update the cell state ctc_tct​ and produce the new hidden state hth_tht​. The entire computation is differentiable, enabling gradient-based training via BPTT.

The key design principle is that the cell state ctc_tct​ is updated additively: old information is scaled (not replaced) by the forget gate, and new information is added (not concatenated). This additive structure creates a gradient highway through the cell state, analogous to residual connections in feedforward networks. When the forget gate is close to 1 and the input gate is close to 0, the cell state passes through nearly unchanged, and the gradient flows backward with minimal attenuation.

ft=σ(Wfxt+Ufht−1+bf)f_t = \sigma(W_f x_t + U_f h_{t-1} + b_f)ft​=σ(Wf​xt​+Uf​ht−1​+bf​)
ftf_tft​σ\sigmaσ

Forget gate activation. Wf∈Rdh×pW_f \in \mathbb{R}^{d_h \times p}Wf​∈Rdh​×p is the input-to-forget weight matrix, Uf∈Rdh×dhU_f \in \mathbb{R}^{d_h \times d_h}Uf​∈Rdh​×dh​ is the hidden-to-forget weight matrix, and bf∈Rdhb_f \in \mathbb{R}^{d_h}bf​∈Rdh​ is the bias. The sigmoid σ\sigmaσ squashes the output to (0,1)(0, 1)(0,1).

it=σ(Wixt+Uiht−1+bi)i_t = \sigma(W_i x_t + U_i h_{t-1} + b_i)it​=σ(Wi​xt​+Ui​ht−1​+bi​)
iti_tit​σ\sigmaσ

Input gate activation. Same structure as the forget gate with independent parameters.

c~t=tanh⁡(Wcxt+Ucht−1+bc)\tilde{c}_t = \tanh(W_c x_t + U_c h_{t-1} + b_c)c~t​=tanh(Wc​xt​+Uc​ht−1​+bc​)
c~t\tilde{c}_tc~t​

Candidate cell state. The tanh activation bounds the candidate to [−1,1][-1, 1][−1,1], preventing unbounded growth of the cell state.

ct=ft⊙ct−1+it⊙c~tc_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_tct​=ft​⊙ct−1​+it​⊙c~t​
ctc_tct​ftf_tft​iti_tit​c~t\tilde{c}_tc~t​

Cell state update. The Hadamard product ⊙\odot⊙ applies element-wise gating. The forget gate scales the old cell state; the input gate scales the new candidate. This is the additive update that preserves gradient flow.

ot=σ(Woxt+Uoht−1+bo)o_t = \sigma(W_o x_t + U_o h_{t-1} + b_o)ot​=σ(Wo​xt​+Uo​ht−1​+bo​)
oto_tot​σ\sigmaσ

Output gate activation. Determines which cell state components are exposed to the network.

ht=ot⊙tanh⁡(ct)h_t = o_t \odot \tanh(c_t)ht​=ot​⊙tanh(ct​)
hth_tht​oto_tot​ctc_tct​

Hidden state output. The tanh squashes the cell state to [−1,1][-1, 1][−1,1] before gating, keeping the hidden state bounded.

Concatenate inputs

The concatenation of the previous hidden state and current input forms the common input to all four gate computations. Some implementations use separate weight matrices for ht−1h_{t-1}ht−1​ and xtx_txt​ (as shown in the equations above); others use a single weight matrix on the concatenated vector.
zt=[ht−1;xt]∈Rdh+pz_t = [h_{t-1}; x_t] \in \mathbb{R}^{d_h + p}zt​=[ht−1​;xt​]∈Rdh​+p
dhd_hdh​

Compute gate pre-activations

Four independent affine transformations computed in parallel. In practice, these are batched into a single matrix multiplication on the concatenated vector ztz_tzt​, then split into four equal chunks.
f^t=Wfxt+Ufht−1+bfi^t=Wixt+Uiht−1+bic^t=Wcxt+Ucht−1+bco^t=Woxt+Uoht−1+bo\begin{aligned} \hat{f}_t &= W_f x_t + U_f h_{t-1} + b_f \\ \hat{i}_t &= W_i x_t + U_i h_{t-1} + b_i \\ \hat{c}_t &= W_c x_t + U_c h_{t-1} + b_c \\ \hat{o}_t &= W_o x_t + U_o h_{t-1} + b_o \end{aligned}f^​t​i^t​c^t​o^t​​=Wf​xt​+Uf​ht−1​+bf​=Wi​xt​+Ui​ht−1​+bi​=Wc​xt​+Uc​ht−1​+bc​=Wo​xt​+Uo​ht−1​+bo​​

Apply nonlinear activations

Sigmoid activations produce values in (0,1)(0, 1)(0,1) for the gates, which act as soft switches. The tanh activation on the candidate produces values in (−1,1)(-1, 1)(−1,1), bounding the magnitude of information written to the cell.
ft=σ(f^t),it=σ(i^t),c~t=tanh⁡(c^t),ot=σ(o^t)f_t = \sigma(\hat{f}_t), \quad i_t = \sigma(\hat{i}_t), \quad \tilde{c}_t = \tanh(\hat{c}_t), \quad o_t = \sigma(\hat{o}_t)ft​=σ(f^​t​),it​=σ(i^t​),c~t​=tanh(c^t​),ot​=σ(o^t​)
ftf_tft​σ\sigmaσiti_tit​c~t\tilde{c}_tc~t​oto_tot​

Update cell state

The additive cell state update. When ft≈1f_t \approx 1ft​≈1 and it≈0i_t \approx 0it​≈0, the cell state is preserved unchanged. When ft≈0f_t \approx 0ft​≈0, old information is erased. The forget gate bias bfb_fbf​ is typically initialized to a positive value (1.0 or 2.0, following Jozefowicz, Zaremba, and Sutskever, 2015) so that the forget gate starts close to 1, encouraging information retention early in training.
ct=ft⊙ct−1+it⊙c~tc_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_tct​=ft​⊙ct−1​+it​⊙c~t​
ctc_tct​ftf_tft​iti_tit​c~t\tilde{c}_tc~t​

Compute hidden state output

The output gate selects which cell dimensions are visible downstream. The tanh ensures the hidden state stays in [−1,1][-1, 1][−1,1] regardless of cell state magnitude.
ht=ot⊙tanh⁡(ct)h_t = o_t \odot \tanh(c_t)ht​=ot​⊙tanh(ct​)
hth_tht​oto_tot​ctc_tct​

Produce forecast from final hidden state

A linear output layer maps the final hidden state hTh_ThT​ to the scalar forecast. For multi-step forecasting, the LSTM can be run autoregressively (feeding y^T+1\hat{y}_{T+1}y^​T+1​ back as input to predict y^T+2\hat{y}_{T+2}y^​T+2​) or trained with a multi-output head.
y^T+h=WyhT+by,Wy∈R1×dh,  by∈R\hat{y}_{T+h} = W_y h_T + b_y, \quad W_y \in \mathbb{R}^{1 \times d_h}, \; b_y \in \mathbb{R}y^​T+h​=Wy​hT​+by​,Wy​∈R1×dh​,by​∈R
dhd_hdh​

Parameter count per layer

Four gates, each with an input weight matrix (dh×pd_h \times pdh​×p), a recurrent weight matrix (dh×dhd_h \times d_hdh​×dh​), and a bias vector (dhd_hdh​). With dh=64d_h = 64dh​=64 and p=20p = 20p=20, this gives 4×64×(20+64+1)=21,7604 \times 64 \times (20 + 64 + 1) = 21{,}7604×64×(20+64+1)=21,760 parameters per LSTM layer.
Nparams=4×[(dh×p)+(dh×dh)+dh]=4dh(p+dh+1)N_{\text{params}} = 4 \times [(d_h \times p) + (d_h \times d_h) + d_h] = 4 d_h (p + d_h + 1)Nparams​=4×[(dh​×p)+(dh​×dh​)+dh​]=4dh​(p+dh​+1)
dhd_hdh​

Why four gates instead of fewer?

The original 1997 LSTM had no forget gate; it was added by Gers, Schmidhuber, and Cummins (2000) after observing that the network needed a mechanism to reset cell state when context changed. The GRU (Cho et al., 2014) reduces the gate count to two by merging the forget and input gates into a single update gate and collapsing cell and hidden state into one vector. Empirical comparisons (Greff et al., 2017) show that the forget gate is the most important component; removing it degrades performance significantly, while simplifying the output gate has minimal effect.

Forget gate bias initialization

Jozefowicz, Zaremba, and Sutskever (2015) found that initializing the forget gate bias to 1.0 or higher dramatically improves LSTM training stability. A bias of 1.0 means σ(1.0)≈0.73\sigma(1.0) \approx 0.73σ(1.0)≈0.73, so the forget gate starts mostly open, preserving cell state by default. Without this initialization trick, the forget gate can start at 0.5, aggressively erasing cell content and preventing the network from learning long-range dependencies early in training.

Transition checks

Agent update
si,t+1=F(si,t,ui,t;θ),Yt=1N∑i=1Ng(si,t)s_{i,t+1}=F(s_{i,t},u_{i,t};\theta),\qquad Y_t=\frac{1}{N}\sum_{i=1}^{N}g(s_{i,t})si,t+1​=F(si,t​,ui,t​;θ),Yt​=N1​i=1∑N​g(si,t​)

The aggregate is computed after individual states are updated. If the rule is nonlinear or stochastic, the aggregate can change abruptly even when the plotted line connects adjacent periods.

Each point is an aggregate after one model update. The connecting line helps the eye, but the law is discrete.

Do not read the line as a continuous differential equation unless one is stated.

Cutoff and contagion
Di,t=1{qi,t>τ},si,t+1=F0(si,t)+Di,tF1(si,t)D_{i,t}=\mathbf{1}\{q_{i,t}>\tau\},\qquad s_{i,t+1}=F_0(s_{i,t})+D_{i,t}F_1(s_{i,t})Di,t​=1{qi,t​>τ},si,t+1​=F0​(si,t​)+Di,t​F1​(si,t​)

The indicator changes value discontinuously at qi,t=τq_{i,t} = \tauqi,t​=τ. Aggregates inherit that discontinuity when enough agents sit near the cutoff.

A cutoff creates a kink or jump because the update rule changes at the threshold.

Do not force a smooth explanation onto a chart section driven by threshold crossing.

Variable timing

Agent state

si,t+1=F(si,t,ui,t;θ)s_{i,t+1}=F(s_{i,t},u_{i,t};\theta)si,t+1​=F(si,t​,ui,t​;θ)

Agent states update at dates. Abrupt aggregate movement can come from many small state changes applied at once.

Aggregate readout

Yt=N−1∑i=1Ng(si,t)Y_t=N^{-1}\sum_{i=1}^{N}g(s_{i,t})Yt​=N−1i=1∑N​g(si,t​)

The aggregate can jump when the distribution changes shape or many agents cross a boundary.

Threshold indicator

Di,t=1{qi,t>τ}D_{i,t}=\mathbf{1}\{q_{i,t}>\tau\}Di,t​=1{qi,t​>τ}

The indicator turns on at the cutoff. That creates a discontinuity in the micro rule and possibly the aggregate.

Proof lineage

Typed relationships across models

Parents -- models this proof draws from

ExtendsGradient BoostingBoth gradient boosting and LSTMs are supervised learning methods optimized via gradient descent on a loss function. Gradient boosting operates in function space (each iteration adds a tree to the ensemble), while the LSTM operates in parameter space (each iteration updates weight matrices via backpropagation). Gradient boosting handles tabular features natively; the LSTM adds the ability to model sequential dependencies through recurrent processing and gated memory.

Keep going

Read next

Continue with a related concept, model, or example.

Theory model

IS-LM Model

a well-connected theoretical model two steps from here

New angle: theoretical model →
Macro by Mark
Help
Privacy PolicyTerms of UseEthics and Compliance

© 2026 Mark Jayson Nation LLC