From Causal Self-Attention (2017) to KDA (2025)

In July 2026, Moonshot AI open-sourced the frontier model Kimi K3 (2.8T parameters). It approached closed-source SOTA models on long-context reasoning and coding. Reading the K3 technical report and open-source code is a useful window into how leading labs design and optimize large language models. This note focuses on one important piece of K3: the linear attention mechanism Kimi Delta Attention (KDA).

KDA is introduced to ease the compute pressure of long context. Full attention costs $O(T^2)$ compute in training as sequence length grows, and the KV cache grows linearly with $T$ at inference. A natural question is whether a cheaper approximation can replace full attention and serve longer context. The idea behind KDA is to compress the KV-cache state, which otherwise grows linearly with context length, into a fixed $d_k \times d_v$ matrix. Starting from causal softmax attention, this note walks through linear attention, DeltaNet, and Gated DeltaNet to KDA, and reconstructs how linear attention evolved.

0. What "Linear" Means

The "linear" in linear attention refers to complexity in the sequence length $T$, not to the model being a linear function. KDA maintains a fixed-size state $S_t\in\mathbb{R}^{d_k\times d_v}$. Each new token requires only one update $S_{t-1}\to S_t$, and the state size does not grow with context length. The networks that produce $q,k,v,\alpha,\beta$ can still include nonlinearities such as activations, convolutions, and normalization.

1. Causal Softmax Attention

In the attention formula, at token position $t$, causal attention only looks at the current position and what came before it. The effective history is the key-value pairs $(k_1,v_1),\ldots,(k_t,v_t)$, and the output is a weighted average of the values:

$$o_t=\sum_{i\le t}\alpha_{ti}v_i.$$

The weight $\alpha_{ti}$ measures similarity by a dot product between the current query $q_t$ and each key $k_i$, then normalizes with softmax:

$$\alpha_{ti} = \frac{\exp(q_t^\top k_i)} {\sum_{j\le t}\exp(q_t^\top k_j)}.$$

In this scheme, the query can randomly access any historical $(k, v)$ pair. The cost is that longer history makes compute and storage more expensive:

$$\text{training compute}\sim O(T^2), \qquad \text{KV Cache}\sim O(T).$$

2. Linear Attention

The key observation in Katharopoulos et al. (2020) is that if the attention kernel $\kappa$ can be factored as

$$\kappa(q,k)=\phi(q)^\top\phi(k),$$

then the attention coefficient $\alpha_{ti}$ can be rewritten as a dot product of two vectors:

$$o_t =\sum_{i\le t}\underbrace{\phi(q_t)^\top\phi(k_i)}_{\alpha_{ti}}\,v_i$$

The next step is to pull out $\phi(q_t)$, which does not depend on history, and gather all history-dependent $(k, v)$ information into one place:

$$o_t ^\top = \sum_{i \le t} \phi(q_t)^\top\phi(k_i)v_i^\top = \sum_{i \le t} \phi(q_t)^\top[\phi(k_i)v_i^\top] = \phi(q_t)^\top \sum_{i \le t} \phi(k_i)v_i^\top$$

Hence:

$$o_t = (\sum_{i \le t} \phi(k_i)v_i^\top)^\top \phi(q_t)$$

Define the state matrix $S_t:=\sum_{i \le t}\phi(k_i)v_i^\top$. The expression simplifies to:

$$o_t = S_{t-1}^\top \phi(q_t)$$

From a linear-algebra point of view, $o_t$ is a linear combination of the row vectors of $S_{t-1}$. We can treat $S_t$ as a finite memory store. By definition, it updates as:

$$S_t \leftarrow S_{t-1} + \phi(k_t)v_t^\top$$

The query now only reads the compressed memory $S_t$, but compute and storage become much cheaper:

$$\text{training compute}\sim O(cT), \qquad \text{Memory Cache}\sim O(1).$$

3. Problems with Naive Linear Attention

The simplest update $S_t=S_{t-1}+k_tv_t^\top$ only accumulates: every new key-value adds an outer product to the state. It does not know whether this information is already stored, whether an old value should be overwritten, or which history has become unimportant. When similar keys keep appearing, different values interfere with each other in the finite state.

We therefore want a rule that can both revise memory and overwrite it.

4. DeltaNet

DeltaNet (Yang et al., 2024) treats the state $S$ as a small online-learned model, and uses the key $k_t$ to predict the corresponding value:

$$\hat v_t=S_{t-1}^\top k_t.$$

Define the reconstruction loss

$$\mathcal L_t(S) = \frac12 \left\| S^\top k_t-v_t \right\|^2,$$

and take one gradient step on $S$ with learning rate $\beta_t$:

$$S_t = S_{t-1} + \beta_tk_t \left( v_t-S_{t-1}^\top k_t \right)^\top.$$

Expanding gives:

$$S_t = (I-\beta_tk_tk_t^\top)S_{t-1} + \beta_tk_tv_t^\top.$$

This is the Delta Rule. The Kimi Linear technical report derives the same update from an online reconstruction loss.

When $k_t$ is normalized, $k_tk_t^\top$ is the projection onto the $k_t$ direction, so the update can be read as: $-\beta_tk_tk_t^\top S_{t-1}$ erases the old association along that direction, and $+\beta_tk_tv_t^\top$ writes the new one. Compared with the vanilla state update, information that is already predicted correctly is not written again; only the residual error enters the state.

5. Gated DeltaNet

The Delta Rule can overwrite the old information associated with a given key, but it cannot actively clear memories that have not been visited for a long time. Gated DeltaNet (Yang et al., 2024) adds a scalar forget gate $\alpha_t\in[0,1]$. It first decays the old state $\widetilde S_{t-1}=\alpha_tS_{t-1}$, then applies the Delta Rule to the decayed state:

$$S_t = \widetilde S_{t-1} + \beta_tk_t \left( v_t-\widetilde S_{t-1}^\top k_t \right)^\top.$$

Expanding gives:

$$S_t = \alpha_t (I-\beta_tk_tk_t^\top)S_{t-1} + \beta_tk_tv_t^\top.$$

$\alpha_t$ controls how much of the whole state is kept; $\beta_t$ controls how strongly the current key direction is overwritten—global forgetting, local erase, and local write.

6. KDA

The limitation of Gated DeltaNet is that each head has only one scalar $\alpha_t$, so every feature channel forgets at the same rate. KDA replaces the scalar gate with a vector $\boldsymbol\alpha_t\in[0,1]^{d_k}$ and defines $D_t=\operatorname{Diag}(\boldsymbol\alpha_t)$.

It first decays channel-wise, $\widetilde S_{t-1}=D_tS_{t-1}$, then applies the Delta Rule:

$$S_t = \widetilde S_{t-1} + \beta_tk_t \left( v_t-\widetilde S_{t-1}^\top k_t \right)^\top.$$

Expanding yields the core recurrence of KDA:

$$\boxed{ S_t=(I-\beta_t k_t k_t^{\top})D_t S_{t-1}+\beta_t k_t v_t^{\top} }$$

The output is $o_t=S_t^\top q_t$.

Per-channel gating gives different dimensions different time scales: some channels forget quickly to track local change, some decay slowly to keep long-term information, and some are actively reset by the current input.

7. Summary

Each KDA step does four things: decay stale memory with $D_t$, predict the value associated with $k_t$ from the current state, compute the prediction error, and write the error back into the state. It is less a variant of an attention formula than a fast-weight model that keeps updating during inference.

Causal softmax attention stores the full history and is suited to precise retrieval. KDA stores a compressed model of history $S_t$, and is suited to ongoing state tracking and working memory in long reasoning, but it cannot guarantee lossless random access to arbitrarily long history. Kimi Linear also does not use KDA alone: it interleaves KDA with global MLA layers at roughly a $3:1$ ratio. KDA handles compression, while global attention preserves random access.