Skip to main content

Cookies on Macro by Mark.We use essential cookies for sign-in, locale, and security. With your permission, we also use optional cookies for preferences, for analytics, and for advertising measurement with the Meta (Facebook) Pixel. Read our cookie policy.

Macro by MarkMacro by Mark
  • Home
Loading…
Loading…
← Models
MacroConceptsPolicyModels

LSTM Model

Recurrent neural network architecture with gates that can retain or forget information across long macro sequences.

OverviewGraphMathReferences

Graph workspace

Capacity and gradient reach of the network

Graph workspace

This page explains what the architecture is, not how one particular fit turned out. Two questions decide whether an LSTM can work on a macro series at all, and both are answered exactly by the design rather than by training: how many parameters the hidden width buys against how few quarterly observations exist to fit them, and how far gradient still reaches once it has been multiplied by a forget gate at every step of the lookback. Running a network and scoring it is a Macro Lab exercise; this graph is for seeing why the answer usually comes out the way it does.

Chart axis metadata
xHidden dimension (capacity panel) / Lag depth (gradient panel)units / steps back · 4 to 128 hidden units; 0 to 60 steps of lookbackyTrainable parameters (capacity panel) / Share of gradient surviving (gradient panel)count / proportion · 101 to 66,689 parameters; 1.0 down to about 10−910^{-9}10−9
Parameters against hidden dimension static chart preview showing Parameters at each width, Parameters left after dropout, Parameters at the chosen width, Training observations available
Gradient retention by lag depth static chart preview showing With the forget gate biased to remember, With no forget-gate bias
Effective update by lag depth static chart preview showing Update reaching this depth, Floor worth calling nonzero
The kind of series this would be
pointed at static chart preview showing Target series at this noise level
Error floor and the benchmarks a
network has to beat static chart preview showing Series 1
xHidden dimension (capacity panel) / Lag depth (gradient panel)units / steps back · 4 to 128 hidden units; 0 to 60 steps of lookbackyTrainable parameters (capacity panel) / Share of gradient surviving (gradient panel)count / proportion · 101 to 66,689 parameters; 1.0 down to about 10−910^{-9}10−9

Current results

Trainable parameters

Explain: Trainable parameters
4(dh2+dhdx+dh)+dh+14\left(d_h^2 + d_h d_x + d_h\right) + d_h + 14(dh2​+dh​dx​+dh​)+dh​+1

Every weight the optimizer has to set. Four gates each carry a recurrent matrix, so the count grows with the SQUARE of the hidden width: 101 parameters at a width of 4, but 66,689 at a width of 128.

4385

count

Effective parameters per observation

Explain: Effective parameters per observation
4(d~2+d~dx+d~)+d~+1T−L,d~=dh(1−pdrop)\frac{4\left(\tilde{d}^2 + \tilde{d} d_x + \tilde{d}\right) + \tilde{d} + 1}{T - L}, \quad \tilde{d} = d_h (1 - p_{\text{drop}})T−L4(d~2+d~dx​+d~)+d~+1​,d~=dh​(1−pdrop​)

Parameters per training example, once dropout has thinned the hidden layer, against the roughly 250 quarters a long macro series offers. Above one there are more free parameters than observations to pin them down. At the default width that ratio is near thirteen, which is the honest reason deep sequence models struggle on macro data.

12.62

ratio

Gradient retention at the deepest lag

Explain: Gradient retention at the deepest lag
∏k=1Lfk≈σ(bf)L\prod_{k=1}^{L} f_k \approx \sigma(b_f)^{L}∏k=1L​fk​≈σ(bf​)L

The share of gradient still alive after being multiplied by a forget gate at every step of the lookback. With the standard bias-one initialization the gate sits near 0.73, so a 24-step window passes back about 5×10−45 \times 10^{-4}5×10−4 of the signal. The gating slows the decay; it does not stop it.

0.0005

proportion

Useful gradient depth

Explain: Useful gradient depth
max⁡{k≤L:η σ(bf)k≥10−6}\max \{ k \le L : \eta \, \sigma(b_f)^{k} \ge 10^{-6} \}max{k≤L:ησ(bf​)k≥10−6}

How many steps back still receive an update worth calling nonzero, taking 10−610^{-6}10−6 as the floor. Compare it with the lookback: when it falls short, the extra window length is costing parameters without teaching the network anything. A larger learning rate pushes the useful depth outward, which is one of the few honest ways to buy reach.

22.00

steps

Controls

balanced
Explain: How complex the model is

Hidden dimension: 32

How much pattern the network can hold; bigger settings need more data to train.

Larger dhd_hdh​: training loss drops faster and lower, validation loss improves initially but may diverge sooner if overfitting. Forecast tracks actual series more closely up to a point.

sees a moderate stretch
Explain: How far back it looks

Lookback window: 24

How many past periods the model sees each time it makes a forecast.

Longer lookback: the network can capture longer cycles, but training slows and overfitting risk increases on short series. Forecast may improve at turning points where long-range context helps.

balanced
Explain: Size of each learning step

Learning rate: 0.001

How much the model adjusts itself at every training step.

Higher learning rate: faster initial loss drop but noisier convergence and earlier overfitting. Lower learning rate: slower, smoother convergence curves requiring more epochs.

lightly restrained
Explain: Share of network switched off

Dropout rate: 0.2

The share of the network switched off at random during training, which curbs memorizing the data.

Higher dropout: training loss stays higher (regularization effect), but the gap between training and validation loss shrinks. Forecast becomes smoother and less reactive to noise.

moderately noisy
Explain: How noisy the data is

Noise level: 0.5

How much random noise sits in the data, hiding the pattern. It changes the chart only; the readouts describe the network's architecture.

Higher noise makes the plotted target series swing more widely. The parameter and gradient panels do not move: the page trains nothing, so there are no loss curves for noise to raise.

Keep going

Read next

Continue with a related concept, model, or example.

Theory model

IS-LM Model

a well-connected theoretical model two steps from here

New angle: theoretical model →
Macro by Mark
Help
Privacy PolicyTerms of UseEthics and Compliance

© 2026 Mark Jayson Nation LLC