Unembedding Proximity Pulls Residual Effects Into Late Layers

Fact-checked by the J-Space editorial team

Updated August 25, 2026

Every token prediction in a decoder-only transformer depends on one shared computational channel from the embedding to the unembedding: the residual stream. Anthropic’s 2021 Transformer Circuits framework treats that channel as an additive communication bus in which each block reads a linear projection of the stream and writes an update, which is why layer-index claims about amplification have to be stated in residual-update algebra rather than as snapshots of a hidden state.

This article clarifies how to define late amplification and how to falsify it in residual stream analysis. It does not claim that early layers lack task information, that a late SAE latent is automatically a louder copy of an early feature, or that residual geometry settles debates about consciousness. The aim is a reporting standard: a named basis, a specified token position, and a layer-wise magnitude or causal-effect curve whose peak still stands after affine readout correction and causal checks. There is a tradeoff in that standard. A claim that cannot survive affine correction and a causal check is easier to publish as a lens plot, and harder to defend as amplification.

The Residual Stream as an Additive Communication Channel

Answer-first: the residual stream is not a hidden snapshot of “what the model is thinking” at a depth index; it is the transformer’s additive bus, and late amplification is a claim about writes on that bus. In decoder-only stacks used throughout residual stream analysis – GPT-2-class models with explicit late-layer stacks of 12-48 blocks on a single residual stream, and the same skip-connected pattern in LLaMA-class and related decoder-only language models – the state at layer ℓ is a vector hℓ in R^d that accumulates attention and MLP writes, plus any layer-normalisation increment the block is allowed to add. Anthropic’s Transformer Circuits framework states the communication rule directly: each block reads a linear projection of the residual stream and writes an update, so the residual-update equation has to be written before any layer-wise plot. With architecture-specific LayerNorm placement made explicit, the Pre-LN form used in GPT-2 and most modern stacks is x_{l+1} = x_l + Attn_l(LN(x_l)) + MLP_l(LN(x_l + Attn_l(…))), while Post-LN (original Vaswani et al. transformer architecture) normalizes after the add. Skip connections – the identity path that preserves x_l – are why information flow is never completely blocked in a single layer even when the learned write is near zero or temporarily misaligned during early training. That design traces to He et al.’s residual networks for computer vision, where identity paths addressed vanishing gradients in deep convolutional neural networks; the same additive principle applies here, with richer dynamics because self-attention mixes tokens. Two contrasts matter for residual stream analysis. First, unlike an RNN hidden state, the residual stream keeps a fixed width at every depth, so earlier writes remain linearly available (though they can be overwritten by later adds). Second, not every basis on that bus is equal: the unembedding rows and the one-hot token embeddings are privileged, as is the LayerNorm scale, in a way the bulk of the residual stream is not. Residual stream activations therefore have to be reported with the hook point named relative to LayerNorm, because a constant-direction write can look larger after a Pre-LN residual-norm trajectory and a final LayerNorm gain stage even when no new decoder-aligned feature was created. The transformer residual stream is the only globally shared channel in the transformer architecture: every head and every MLP must perform addition into the same workspace, which is why contributions from different layers can be attributed at all.

x_{l+1} = x_l + Attn_l + MLP_l   # residual stream at the next block equals the current residual stream plus the attention write plus the MLP write; LayerNorm placement is architecture-specific and must be stated at the hook

Why “late” is a write-index claim, not a claim that earlier layers lack the information, follows from the same algebra. Only late writes sit immediately upstream of W_U, so decoder-aligned effects are mechanically biased toward the last blocks unless earlier writes remain in that basis all the way to the unembedding. Mid-depth residual stream activations can already contain entity features or induction copies; residual stream analysis still has to ask whether those directions were amplified along a fixed decoder-aligned axis or only later rotated into vocabulary space. Local linearization of the residual stream map, Jℓ = ∂h_{ℓ+1}/∂hℓ, decomposes as Jℓ = I + Rℓ because of the skip path: I is the identity contribution and Rℓ is the Jacobian of the learned block write. Cumulative products Pℓ = J_{L-1} · … · Jℓ then describe end-to-end sensitivity of the final layer to a perturbation at depth ℓ. That mathematical framework is a probe of information flow, not a proof that spectral radius alone is amplification. Non-normal (off-diagonal) structure can produce transient growth even when every eigenvalue magnitude is at most one; a Schur split Jℓ = Qℓ (Λℓ + Nℓ) Qℓ* separates the diagonal eigenvalue part from the strictly upper-triangular non-normal part that lets directions borrow energy across adjacent layers. Self-alignment (overlap of leading input and output singular subspaces) and Henrici departure from normality are legitimate diagnostics of rotation versus stretch, provided they are not treated as capability scores. Privileged-basis evidence from residual stream activations – high kurtosis and structured tails in the native coordinates, versus nearer-Gaussian statistics after a random rotation – matters because late layers can cheaply stretch a small subset of already-aligned coordinates instead of learning a new rotation into W_U. Residual streams encode complex geometric structure at more than one scale, which is part of why dictionary learning and feature-based steering are usually applied in the residual stream rather than in an arbitrary internal basis. None of this requires asserting an unpublished eigenvalue percentage or an unpublished effective-rank integer; it requires stating the residual-update algebra and the hook before interpreting any late peak.

What separates late amplification from late decoding?

Answer-first: residual stream analysis treats the trajectory {hℓ} as a path through one shared space and asks which directions are stretched toward the decision, not merely which tokens become linearly decodable. The operational definition has two necessary measurement pieces plus a causal bar. First, a direction or latent with a named basis (neuron, SAE decoder column, PCA axis, or unembedding row). Second, a layer-wise magnitude or causal-effect curve on residual stream activations at a stated token position (final token, answer token, or induction site). The late peak then has to survive causal tests rather than appearing only on a passive readout. Without the token position the signature is uninterpretable, because the same layer index can mean context-gathering at one site and commitment at another. Residual stream analysis is therefore a claim about information flow across multiple layers, not a synonym for “the logit lens looks more like the answer later.”

The distinction that most current explainers skip is amplification along a fixed decoder-aligned direction versus rotation of a mid-layer feature into W_U. A vocabulary readout can improve because the residual stream was rotated into the unembedding basis even if the feature’s intrinsic amplitude did not grow. Belrose et al.’s Tuned Lens exists specifically because raw intermediate residuals are not in the unembedding basis; any serious amplification claim has to survive that affine correction. Late amplification can distinguish when information is formed from when it gains decision weight: a fact may be present from an early block onward and only converted to a token-aligned write near the final layer. That is the operational gap between “present in the residual stream” and “amplified into the decision.” The counterargument is that practitioners often need late decoding, not amplification, because the unembedding only sees the rotated residual. That quantity is useful. It answers a different question. Residual stream analysis that reports only a prettier late logit-lens plot has documented late decoding until it also reports a fixed-direction curve and a causal check against a negative-control direction.

Which measurements support an amplification claim?

Answer-first: a measurement stack for residual stream activations has to show norms, a readout, and a write-level attribution on the same token position, or the late peak cannot be interpreted. Residual L2 and RMS trajectories through depth, before and after each LayerNorm, are the first plot, not the last, because Pre-LN residual-norm growth makes a constant-direction write look larger after the final LayerNorm. Researchers often track vector energy in the residual stream with the ℓ2 norm as a complement to any spectral probe. The readout stack then splits three ways. Logit lens applies W_U (after a final LayerNorm) directly to intermediate residual stream activations; a late rise is expected and is not, by itself, amplification. Tuned lens and related future-lens maps fit an affine translator from each intermediate state into the unembedding basis, which is the standard correction when the claim is that a prediction was amplified rather than rotated into vocabulary space. Direct logit attribution (DLA) dots attention-head and MLP writes into W_U so that each write’s contribution to a target logit is visible without assuming that the entire residual stream is already in vocabulary coordinates. Sparse autoencoders add a fourth view: they factor residual stream activations into an overcomplete dictionary so that a latent’s norm trajectory can be plotted on the same axes as DLA. Activation patching and layer-localized steering then test whether intervening on that direction at that depth changes the output. Linear probes can show that a feature is decodable and still miss non-normal, depth-dependent information flow. Attention-weight plots are not information flow. Neuron-level inspection of a single layer can be useful in toy models and still fail to scale once units are polysemantic. The residual stream remains the preferred locus because it is the shared channel and because additive writes let residual stream analysis cleanly separate contributions that other internal tensors mix. On the other hand, stacking every probe on every prompt is expensive, and many papers reasonably stop at tuned lens plus DLA. The cheaper stack is also the more confounded one unless Pre-LN growth is reported beside the readout.

Confounds that impersonate amplification have to be listed beside the stack, or residual stream analysis will over-claim. Final LayerNorm plus unembedding is itself a gain stage; pre- and post-LN curves are both required. Superposition and dictionary drift mean a late SAE latent may be a different feature, not a louder copy of an early one, especially when a single layer dictionary is compared naively with dictionaries on adjacent layers. Average Jacobians over mixed text input can erase context-specific peaks. Local Jacobians are linearizations around typical residual stream activations; large or adversarial perturbations can engage nonlinear structure those matrices miss. Most public analyses still condition on one token position, usually the last decoding position, which is a legitimate hook and not a sequence-level law. When Jacobian products are used, effective rank of Pℓ is a diagnostic of how many directions still affect the final layer, not a pruning score and not a substitute for DLA. Spectral radius of a single layer Jacobian can correlate only weakly with the cumulative product, which is why residual stream analysis should not reduce amplification to one eigenvalue. A practical recipe is then: freeze the model; collect residual stream activations on stated text input; choose the token position; compute norms around LayerNorm; run logit lens and tuned lens; attribute writes with DLA; optionally project sparse autoencoders or a multi layer SAE dictionary through the same curve; then patch or steer by layer against a negative-control direction. That is residual stream analysis as an audit procedure rather than as a gallery of plots.

Multi Layer SAE Work, Dictionary Learning, and Interpretable Features

Answer-first: dictionary learning on the residual stream is how residual stream activations become named latents, and a multi layer SAE is how those latents are tested for persistence across multiple layers instead of being invented independently at each depth. Standard sparse autoencoders trained on a single layer of residual stream activations have been the workhorse for decomposing superposition into overcomplete, often monosemantic directions. They do not, by themselves, say whether a feature persists, transforms, or is replaced on adjacent layers. A multi layer SAE trains on residual stream activations from more than one depth at once – for example several mid and late blocks in the same transformer residual stream – and learns a shared dictionary that must reconstruct residual stream activations at all included depths. Larger models often show higher cosine similarity of residual stream activations across adjacent layers, which is consistent with more features persisting across depth and is one reason a multi layer SAE is worth training rather than stitching unrelated single layer dictionaries. Features from a multi layer SAE that light up in both mid and late residual stream activations are natural candidates for directions that might experience late amplification, but candidacy is not a causal effect size: the multi layer SAE latent still has to be projected through DLA or through a layer-wise steering/ablation curve. The tradeoff in a multi layer SAE is reconstruction coverage versus feature identity. A shared dictionary can force one code to stand in for two different latents, which would make a late-norm rise look like amplification when it is dictionary compromise. Training can change how clean residual stream activations look to sparse autoencoders; features that are blurry early in optimization often become easier to read as the model’s late writes become more output-aligned. A multi layer SAE can also capture gradual onset that a single layer dictionary misses, particularly structure that accumulates across subsequent layers rather than appearing as a step at one index. The practical filter is then: keep multi layer SAE directions whose decoder columns align with the late, unembedding-aligned subspace, and drop multi layer SAE directions that reconstruct residual stream activations without moving the target logit. That is how residual stream analysis connects the “what” of dictionary learning to the “how” of information flow without treating every interpretable features list as a circuit.

Public SAE studies on residual streams are the primary source for the late-versus-early content split. Anthropic’s dictionary-learning work on production-scale residual streams is the main public evidence that late-layer latents are more output-aligned and more safety-relevant, while early latents are more contextual. Sparse autoencoders extract interpretable features from residual stream activations by reconstructing those activations as sparse combinations of dictionary columns. Reported feature families include geographic and entity structure, code-vulnerability patterns, and deception or sycophancy-related directions. Those labels are hypotheses about residual stream activations until ablation, clamping, or steering shows a causal effect. Attribution studies in this literature often find that some specific features – especially planning and policy-related latents – become more decision-relevant in late residual stream activations even when the same dictionary element is already nonzero earlier. A multi layer SAE is the tool that makes that “already nonzero” claim checkable across multiple layers with one set of codes. Combining a multi layer SAE with DLA on the same axes is the competitor gap most pages miss: without both, output-aligned latents get narrated as circuits and never receive an effect size. Sparse autoencoders also interact with superposition: many features can share neurons in residual stream activations, so a late rise in one multi layer SAE code is not automatically a louder copy of an early code with a similar autointerp caption. Dictionary drift across different layers is a falsifier. If a multi layer SAE cannot reconstruct both depths without splitting the feature, residual stream analysis should not say “amplified.” If it can, the next plot is still causal. Sparse autoencoders are not a substitute for tuned lens or DLA; they name directions those probes can then score. Residual stream activations remain the preferred training site for sparse autoencoders because the transformer residual stream is the additive workspace every write must use, which is why a multi layer SAE on that workspace is a more direct window on information flow than a multi layer SAE on an arbitrary MLP activation tensor.

Where late signatures are actually observed is narrower than “somewhere near the end of the network.” Next-token commitment and entropy collapse often concentrate in the last block of layers, which is expected once residual stream activations are being read by W_U. Induction and other in-context-copy circuits write a late, unembedding-aligned name-mover signal into the residual stream at a specific token position; that is a canonical late write rather than a mid-depth contextual feature. Factual-recall and multi-hop items often show mid-layer entity structure in residual stream activations that is only converted to tokens late, which is exactly the present-versus-amplified distinction activation patching has been used to make. Semantic boundaries – paragraph breaks, topic shifts, formatting changes – are another site where residual stream activations must resolve competing continuations. Residual stream trajectories can bend at those semantic boundaries, recruiting MLP writes and inducing curvature that is easy to misread as generic “late complexity.” Expect boundary-adjacent token position curves to look different from curves inside homogeneous text input: more local directions in play, stronger funneling toward whatever write resolves the boundary. None of those observations requires a global-workspace theory of mind. The residual stream is a candidate shared workspace only in the limited sense that many modules read and write one bus and that late writes broadcast a small set of directions to the unembedding and, via the KV cache, to later tokens. Residual streams can be described as carrying a compressed running state of prior text input; that is an information-flow description. Whether that pattern is consciousness is not a question residual stream analysis answers. Privileged-basis structure does, however, explain a narrower fact: once residual stream activations align semantically meaningful directions with coordinates more than a random basis would, late blocks can stretch those coordinates instead of learning a fresh rotation. That is a mechanism for why steering and dictionary learning work better in the residual stream than in a rotated copy of the same residual stream activations.

Jacobian- and community-level residual stream analysis should be treated as additional measurement layers, not as a second paper’s uncited leaderboard. Per-layer Jacobians on residual stream activations, their singular values, and cumulative products are legitimate ways to ask how many directions still affect the final layer. Effective rank of those products is a way to talk about dimensional funneling: as layer distance from the unembedding shrinks, fewer directions retain large singular values. That funnel is a hypothesis about information flow through multiple layers, and it must be shown with a negative-control direction and a causal curve, not only with a spectrum. Non-normality – operators that can produce transient growth because left and right singular vectors fail to coincide – is a reason eigenvalues alone are insufficient. A dose-response that scales only the non-normal Schur block, holding eigenvalues fixed, is a valid internal falsifier if it is actually run and reported; it is not a result this article invents. Activation-correlation graphs on residual stream activations (units as nodes, strong pairwise correlations on text input as edges, community detection for mesoscale clusters) can locate boundary units with high participation coefficients. If those units are preferentially written to in late residual stream activations, that is a coupling worth testing causally; community labels are not by themselves interpretable features. Centered residual-block weight fingerprints are a separate, weight-space question about training lineage and fine tuning. They may complement runtime residual stream activations, but they are not a substitute for a late, position-specific causal curve. Across model families, the qualitative idea that early blocks mix and later blocks commit is compatible with GPT-2-class depth and with the public SAE split between early context latents and late output-policy latents. Architecture still moves the clock: Pre-LN versus Post-LN, parallel residual blocks, and gated MLPs all change when a write becomes decoder-aligned, as do mixture-of-experts writing sparsely into a shared residual stream. Depth itself can push commitment later even when information is already present. Norm plots are not portable across those choices. Residual stream analysis that copies a GPT-2 layer index onto a deeper Pre-LN stack is misreading the bus.

Information Flow, Scaling Laws, and Control Surfaces in Language Models

Answer-first: layer-localized steering and rank-one ablation only make mechanistic sense if the signature is late, decoder-aligned, and pinned to a token position on the residual stream. A steering vector added into residual stream activations is a write. If the true policy computation lives in earlier residual stream activations and is merely read out late, a late steer can look universal while remaining a readout edit, and earlier blocks can compensate. That is why a single “refusal direction” can appear to transfer across prompts and still be a late-layer readout rather than the whole policy. Safety posts that treat refusal or sycophancy vectors as static model properties miss this: they are often late, position-specific writes into the residual stream and can be compensated by earlier writes under distribution shift. The counterargument to late-layer steering as a safety control is that compensation. If earlier residual stream activations still compute the policy, a late write is a readout edit and will not survive a determined backup path. However, a late refusal direction can still be the cheapest audit hook even if it is not the whole policy; the cost is that an earlier compensatory write will look like a failed control. Failure modes are already familiar from circuit work. Downstream compensation and Hydra-like backup heads can restore a behavior after a single layer edit. Fine tuning and instruction tuning can move the decoder-aligned subspace enough that a residual-stream direction named on the base model no longer has the same DLA sign. Residual stream analysis for safety should therefore report the hook and token position, plus the basis (often a multi layer SAE latent or an unembedding-aligned difference-in-means direction), and a pre-registered falsifier: a control direction and at least one earlier-layer intervention that should not fully kill the behavior if the claim is truly late. NIST’s AI Risk Management Framework lists interpretability and explainability among core trustworthiness characteristics; that is the reason to file these residual-stream curves as audit artifacts. It is not a reason to claim that monitoring residual stream activations is a complete safety system. Near-term use is diagnostic: if a known harmful latent’s late gain on residual stream activations spikes during fine tuning, that is a signal for a slower, causal review, not an automatic deletion of a subspace.

Scale, depth, and training stage change timing without cancelling the additive-bus logic. Deeper stacks can keep more computation in the middle residual stream and commit later. Wider stacks can support more features in residual stream activations and still concentrate decision-relevant writes. Mixture-of-experts models share a residual stream while allowing sparse expert writes that act as late, conditional amplifiers; those writes still have to be attributed at a hook and a token position. Classic loss scaling laws relate loss to compute and parameter count. Structural scaling laws for residual stream activations – how many decoder-aligned directions grow, or how sharply tuned-lens commitment moves – are research questions, not fitted exponents in this article. Sparse autoencoders on residual streams of larger models need more features to reconstruct residual stream activations, but the number of features that actually receive late DLA mass need not grow as fast as width if information flow funnels. SAE training loss is a reconstruction metric, not an amplification metric; a wider dictionary can capture more features in the residual stream and still miss the causal subspace. Residual stream analysis should therefore not treat “we fit more features” as “we found more amplifiers.” Training data composition and fine tuning both move residual stream activations. Instruction tuning can install or relocate refusal-related writes. Domain shift (code versus prose) can change which residual stream activations are late-amplified at semantic boundaries. Initialization typically lacks a stable late, decoder-aligned policy write; training installs output-aligned structure as the language-modeling objective forces the residual stream to be readable by W_U. That qualitative statement does not require an unpublished step index. What it does require is checkpointed residual stream analysis if the claim is about installation rather than about a finished model.

How to report a residual-stream amplification claim is a methods standard, not a style preference. Required figures are a norm curve of residual stream activations (pre- and post-LN), a lens or DLA curve, a causal patch/ablation/steering curve by layer, and a negative-control direction that should not show the same late peak. Always name the basis (neuron, multi layer SAE latent, PCA, or unembedding row) and the training or inference objective that produced the residual stream. Always name the token position. Always name the hook relative to layer normalisation. State whether the residual stream is folded (LayerNorm and adjacent linear maps combined) or raw. Reproducibility needs model ID, hook points, and the exact text input family. How to read a layer-wise signature plot follows from those labels. The x-axis is depth, not time, and is not comparable across Pre-LN and Post-LN stacks without a note. The y-axis is a named metric: RMS of residual stream activations, tuned-lens target logit, DLA on a write, or a multi layer SAE coefficient. A late rise on logit lens only is late decoding until the other panels agree. A late rise on a multi layer SAE coefficient without DLA is a dictionary event. A late causal effect without a fixed-direction amplitude curve may be a rotation into W_U. Global properties of information flow – for example that many features are transported in parallel in early residual stream activations while late residual stream activations filter – belong in the caption only if the plot actually shows them. Understanding neural networks at this grain is understanding neural networks as residual dynamical systems, not as a single-layer lookup. Compared with linear probes, Jacobian or DLA views of the residual stream speak to both local writes and information flow across multiple layers. Compared with attention visualization, they track residual stream activations rather than softmax weights. The transformer residual stream is the right object because the transformer architecture forces every submodule to perform addition into it. Transformer circuits research is the right citation style because it already writes that algebra. Residual stream analysis that omits the algebra will keep rediscovering that the last readout looks like the answer.

Implications and limits should stay inside what the measurement can support. Architectural knobs – depth-to-width and LayerNorm placement – change residual stream activations and can move late writes; they are not, on present evidence, a recipe for installing a desired amplifier. Regularizers on cumulative rank or on late DLA mass are conceivable and untested here. Aggressive suppression of late residual-stream gain might affect calibration or long-horizon writes; that is an empirical risk, not a recommendation. Jacobian-aware optimizers that track per-layer residual-stream maps are a research idea, not a training standard. Open questions include whether late decoder-aligned writes are necessary for strong next-token performance, and whether there are prompt families where important residual stream activations are under-amplified. How prefill versus decode changes the same token position, and how stable amplified directions remain under fine tuning and domain shift, are likewise unsettled. Adversarial training regimes that deliberately move late writes would also limit forensic use of residual-stream fingerprints. Multi-token and sequence-level residual stream analysis remains thinner than last-token work. Average curves on mixed text input can hide the semantic boundaries where residual stream activations actually commit. The defensible position is modest and mechanistic: the transformer residual stream is an additive bus; tuned lens exists because intermediate residual stream activations are not born in W_U; public sparse autoencoders find more output-policy latents late; causal practice already separates “present” from “amplified into the decision.” Residual stream analysis that reports those four facts with hooks and falsifiers is ready to be cited. Residual stream analysis that only says later layers look more like the answer is late decoding, and it should be labeled that way.

Editorial Team

Editorial Team

The J-Space Editorial Team curates and synthesizes published research in mechanistic interpretability and AI alignment. With a commitment to empirical rigor and clear exposition, the team distills complex technical developments into accessible yet authoritative analyses for researchers and engineers working at the frontier of artificial intelligence.