Cookies on Macro by Mark.We use essential cookies for sign-in, locale, and security. With your permission, we also use optional cookies for preferences, for analytics, and for advertising measurement with the Meta (Facebook) Pixel. Read our cookie policy.
Recurrent neural network architecture with gates that can retain or forget information across long macro sequences.
Follow the same equations and assumptions one section at a time or in a continuous reading.
Hover or focus a symbol defined in this model to find its exact matches. Select it to keep the highlight; Escape clears the selection.
The LSTM forward pass at each time step computes four quantities in parallel from the concatenation of the previous hidden state and the current input : the forget gate , the input gate , the candidate cell state , and the output gate . These gating values then combine to update the cell state and produce the new hidden state . The entire computation is differentiable, enabling gradient-based training via BPTT.
The key design principle is that the cell state is updated additively: old information is scaled (not replaced) by the forget gate, and new information is added (not concatenated). This additive structure creates a gradient highway through the cell state, analogous to residual connections in feedforward networks. When the forget gate is close to 1 and the input gate is close to 0, the cell state passes through nearly unchanged, and the gradient flows backward with minimal attenuation.
Forget gate activation. is the input-to-forget weight matrix, is the hidden-to-forget weight matrix, and is the bias. The sigmoid squashes the output to .
Input gate activation. Same structure as the forget gate with independent parameters.
Candidate cell state. The tanh activation bounds the candidate to , preventing unbounded growth of the cell state.
Cell state update. The Hadamard product applies element-wise gating. The forget gate scales the old cell state; the input gate scales the new candidate. This is the additive update that preserves gradient flow.
Output gate activation. Determines which cell state components are exposed to the network.
Hidden state output. The tanh squashes the cell state to before gating, keeping the hidden state bounded.
Why four gates instead of fewer?
The original 1997 LSTM had no forget gate; it was added by Gers, Schmidhuber, and Cummins (2000) after observing that the network needed a mechanism to reset cell state when context changed. The GRU (Cho et al., 2014) reduces the gate count to two by merging the forget and input gates into a single update gate and collapsing cell and hidden state into one vector. Empirical comparisons (Greff et al., 2017) show that the forget gate is the most important component; removing it degrades performance significantly, while simplifying the output gate has minimal effect.
Forget gate bias initialization
Jozefowicz, Zaremba, and Sutskever (2015) found that initializing the forget gate bias to 1.0 or higher dramatically improves LSTM training stability. A bias of 1.0 means , so the forget gate starts mostly open, preserving cell state by default. Without this initialization trick, the forget gate can start at 0.5, aggressively erasing cell content and preventing the network from learning long-range dependencies early in training.
The aggregate is computed after individual states are updated. If the rule is nonlinear or stochastic, the aggregate can change abruptly even when the plotted line connects adjacent periods.
Each point is an aggregate after one model update. The connecting line helps the eye, but the law is discrete.
Do not read the line as a continuous differential equation unless one is stated.
The indicator changes value discontinuously at . Aggregates inherit that discontinuity when enough agents sit near the cutoff.
A cutoff creates a kink or jump because the update rule changes at the threshold.
Do not force a smooth explanation onto a chart section driven by threshold crossing.
Agent states update at dates. Abrupt aggregate movement can come from many small state changes applied at once.
The aggregate can jump when the distribution changes shape or many agents cross a boundary.
The indicator turns on at the cutoff. That creates a discontinuity in the micro rule and possibly the aggregate.
Proof lineage
Typed relationships across models