In Arora, Wu, Steinhardt, and Schwettmann, arXiv:2601.22594v2 (§3.1), the residual stream after layer i is the running sum of block outputs so far: r(i) = r(i−1) + a(i) + m(i), with r(0) = e. Here a(i) is the multi-head attention output and m(i) is the MLP-block output. Both are length-n sequences of vectors in Rdmodel, as is r(i). When a circuit is discussed over representations from embeddings through logits, that paper labels the input embeddings e as r(0) and the output logits y as r(L+1). The logit label is a convenience, not another residual addend: each yj is in Rdvocab. Pre-down MLP activations h(i) are not terms in the sum. Superscripts below are the conventional reading of parenthetical layer indices in the extracted PDF text.
Circuit graphs that run from token embeddings to next-token logits need a layer-indexed state those edges can sit on. In arXiv:2601.22594v2 §3.1, that state is the residual stream: at each token position, the running sum of component outputs so far, updated by adding the attention-block output and the MLP-block output.
This note states only the running-sum definition used for that span in arXiv:2601.22594v2 (PDF stamp arXiv:2601.22594v2 [cs.CL] 11 Jun 2026; listed as ICML 2026, PMLR 306). It is notation from the paper’s circuit-tracing preliminaries, not a measurement on a named checkpoint, width, or intervention.
The indexed definition has four constraints that matter once circuits are drawn over these states.
- After layer i ∈ {1, …, L}, r(i) = r(i−1) + a(i) + m(i), and r(0) = e.
- The addends are block outputs in Rdmodel: one multi-head attention output and one MLP output per token, not pre-down activations and not separate head vectors.
- For circuits over these representations, embeddings are r(0) and logits are labeled r(L+1). Logits are not written as a further residual sum.
- Unrolling the recurrence gives r(L) = e + Σi=1L (a(i) + m(i)) under sequence addition. That expansion is a consequence of the stated recurrence, not a separately reported equation.
In This Guide
The residual after layer i is the previous residual plus both block outputs
Section 3.1 of arXiv:2601.22594v2 defines a Transformer language model M with L Transformer blocks. Each block has a sequence-mixing attention block followed by a state-mixing MLP block, with residual connections. The input is a token sequence x = (x1, …, xn), embedded as e = (e1, …, en) with ej ∈ Rdmodel. No numeric dmodel, dffn, or dvocab is part of the definition.
At layer i, the paper defines the attention output, the pre-down MLP activations, and the MLP output, and only then the residual update. The attention output a(i) = (a(i)1, …, a(i)n) has a(i)j ∈ Rdmodel, the output of multi-head attention. The MLP output m(i) = (m(i)1, …, m(i)n) has m(i)j ∈ Rdmodel, the output of the MLP block. Those two block outputs, not the residual itself, are the new terms in the update. The residual stream r(i) = (r(i)1, …, r(i)n), with r(i)j ∈ Rdmodel, is the state after that update. The extracted text calls each r(i)j “a running sum of outputs of all components so far” and writes r(i) = r(i−1) + a(i) + m(i) with r(0) = e.
So the object indexed by layer is the state after both writes of that block have been added to the previous residual. The prose order is attention, then MLP. Section 3.1 does not introduce a separate symbol for a post-attention, pre-MLP residual. It also does not write a LayerNorm or RMSNorm operator into this recurrence. Later sections of the same paper linearize RMSNorm for a replacement model used in attribution; that approximation is not a redefinition of r(i).
Because r, a, and m are length-n sequences of equal dimension, the displayed sum is position-wise. Cross-token mixing, if any, sits inside the attention block that produced a(i), not in the residual addition itself. Iterating the recurrence from r(0) = e yields r(L) = e + Σi=1L (a(i) + m(i)). That is only the sum the definition already states, unfolded. It is not an extra empirical identity, and it does not specify how a(i) or m(i) read their inputs beyond the block order given in the same section.
Pre-down MLP activations are not residual addends
The same preliminaries define MLP activations h(i) = (h(i)1, …, h(i)n) with h(i)j ∈ Rdffn as the pre-down-projection hidden activations inside the MLP. Those coordinates are a different space from the residual. The residual update adds m(i), the MLP-block output in Rdmodel, not h(i).
That split is what makes the running sum well-typed. r(i−1), a(i), and m(i) share dmodel. A pre-down activation cannot be added to them without a map the definition does not include. In the paper’s circuit definition, nodes are computational units such as MLP neurons h(i)j,v or SAE features, and the same unit at different token positions is a distinct node. Those nodes can be discussed over residual representations, but a neuron activation is not itself a term of r(i). Attention is likewise one block output a(i)j per position in this sum, not a list of per-head writes.
Embeddings are r(0); logits are labeled r(L+1) only as a circuit index
After the block recurrence, the model produces logits y = (y1, …, yn) with yj ∈ Rdvocab. Section 3.1 then states the indexing used when circuits span those representations. The sentence in the extracted text is: “For convenience, we refer to the input embeddings e as r(0) and the output logits y as r(L+1) when discussing circuits over these representations.”
Two boundaries follow from that wording. First, r(0) = e is the base case of the running sum, so the embedding sequence is both the input representation and the residual before any block write. Second, r(L+1) is a label for y, introduced “for convenience” when the circuit discussion includes logits. The section does not define r(L+1) = r(L) + (some unembedding write), and it cannot: y is not a dmodel sequence. Treating logits as one more residual vector would change the type of the index. The running sum that is actually defined stops at r(L), the sum of embeddings and every attention and MLP block output through layer L.
What this definition does not say
The recurrence is an authorial definition of the representation, not a causal intervention, a faithfulness result, or a claim about which coordinates are interpretable. It does not name a model, checkpoint, layer width, or ablation. Nothing in §3.1 fixes whether an intermediate activation after attention alone is stored; only r(i), after both block outputs, is indexed.
A different computational graph should not be substituted for this sum. Chen, Plaat, and van Stein, arXiv:2605.23393v3 (PDF stamp 24 Sep 2026) also treat the residual as additive, citing Elhage et al. (2021), but their extracted statement is a sum over component writes at a position: Xj = Σk ckj, where each ckj is an attention-head output, an MLP output, or the input embedding. That is finer than one attention-block vector a(i)j, and the retrieved text does not label logits as r(L+1). Their later attribution also treats LayerNorm scale as constant so a marginal decomposition stays additive, under a pre-norm assumption for GPT-2 and Pythia. Those are extra assumptions. They are not part of the Arora et al. running sum.
Equation glyphs in both PDFs are flattened in the text used here. The reading above takes parenthetical indices as the layer and token indices required by the surrounding prose (i ∈ {1, …, L}, sequences of length n, r(0) = e). It does not recover an unstated unembedding formula, a per-head split of a(i), or a numeric hidden size.
How We Sourced This
Documents cited in this article: arXiv:2601.22594v2, arXiv:2605.23393v3.
- The residual after layer i is the previous residual plus both block outputs
- Pre-down MLP activations are not residual addends
- Embeddings are r(0); logits are labeled r(L+1) only as a circuit index
- What this definition does not say
Sources
FAQ
Does the layer-i residual include the MLP’s hidden activations?
No. In arXiv:2601.22594v2 §3.1, h(i) denotes pre-down-projection activations in Rdffn. The residual update adds the MLP-block output m(i) in Rdmodel, together with the attention-block output a(i), to r(i−1).
Is r(L+1) still a residual-stream vector?
Not under the definition as written. The paper uses r(L+1) as a convenience name for the logit sequence y when circuits are discussed over embeddings and logits. Each logit vector is in Rdvocab. The running sum that stays in residual space is r(L) = e + Σi=1L (a(i) + m(i)).
