A Scale–State Feedback Governs Recurrent-State Quantization in Linear-Attention and SSM LLMs
Independent Researcher
NeurIPS 2026 Workshop on Long-Context Foundation Models
Linear-attention and state-space LLMs carry long context in a fixed-size recurrent state, and quantizing that state is the natural way to cut its cost during long decode. Unlike a weight tensor, the state is rewritten at every step, so a quantizer applied at each step sits inside the recurrence and its errors compound. INT4 that leaves synthetic recall perfectly intact at the start destroys it a thousand steps later. The mechanism is a feedback between the quantizer scale and the state it is derived from. Yoking that scale to a full-precision reference removes the resulting norm explosion entirely, yet recovers only 1.8 of the 5.4 nats lost, so numerical instability and functional degradation are separable failures. The remainder is a resolution problem that locating outliers does not solve, since every outlier-aware scheme here loses to plain persistent scaling. Allocating bits across layers does help, but under article-disjoint splits a learned allocation collapses onto the trivial rule of protecting the first few layers. Run out to 32K, group-wise INT4 is worth no more than discarding the state. Contain the scale rather than locate outliers, and report the decode horizon at which a bit-width was certified.
Quantize the recurrent state at every decode step and ask the model, at the end, for a fact planted before the quantization began. At the moment quantization starts, every bit-width answers perfectly. Drag the horizon and watch where each one lets go.
Per-token dynamic INT4 inflates the state by two to three orders of magnitude. Every static curve sits on the unquantized reference.
Nothing is measured between H=0 and H=1024, so the page draws that span faintly, as the paper does.
Per-token dynamic scaling reads its scale from the state it is about to quantize. Quantization error inflates the state; a bigger state raises the scale; a coarser grid inflates it further. To test whether that loop is the cause rather than a correlate, we run the same quantizer twice and, in one arm, take the scale from a full-precision reference running alongside — the same grid, the same bit-width, but the loop cut.
GDN 0.8B, 16 held-out articles, H=2048. The 2B model behaves the same way; both are in data.js.
If the loop runs through the scale, anything that stops the scale from tracking the state should help — and what helps says what the loop is made of. Pick an intervention.
Every scheme below is given more information than uniform group-wise scaling, at a matched or higher bit rate. None of them beats it, and giving the extra bits to the most-overflowing channels does not beat giving them to a random subset.
Yoking removes the runaway and leaves most of the loss. Two accounts survive that: the grid is too coarse, or rounding error simply accumulates over thousands of steps. They make opposite predictions about bit-width, so we ran it — with the scale yoked, so the feedback cannot confound the sweep.
Every policy here is certified on natural text at the horizon you choose. Choose a short one and the verdict is encouraging. The same runs continue to 32K.
The recurrent state is a fixed cost; in a hybrid, the attention KV cache next to it grows with context. So the case for quantizing the state is strongest in a pure SSM, and weakest exactly where long-context arguments live. Computed from the published configs.
This is the paper's own limitation, and it is in the Limitations section rather than buried.
The frontier and the feedback do. Which containment wins does not. Below are the same four policies under the same protocol — 16 held-out articles, H=2048 — on two delta-rule models and one SSM. Read each row against its own dashed control: a policy to the right of it costs more than discarding the state outright.
Non-archival workshop paper. Update this entry if an archival version appears.