Confirm Layer Selection With Patching Before You Trust Attention Maps

Fact-checked by the J-Space editorial team

Updated August 17, 2026

Transformer-based language models are still often described as black boxes whose internal work is a bag of static features. That picture was never quite right. A more accurate computational description is that transformer layers implement emergent selection over hypotheses in the residual stream, a view already implicit in the original architecture paper and now documented in circuit-level programs such as Anthropic Transformer Circuits. Cataloging neurons or plotting attention weights is not the same as naming what was selected on a given forward pass.

This article clarifies what selection means in attention, MLP, and residual-stream terms, why attention maps are an incomplete explanation, and the implications of those selection processes for interpretability practices. It does not claim that full mechanistic interpretability is imminent or impossible, and it stays neutral on consciousness debates. Numbers and quotations below are limited to named primary sources; unverified layer-fraction or capacity figures are not repeated as facts.

What Emergent Selection Is Inside Transformer Layers and the Residual Stream

Selection inside a transformer layer is learned competitive allocation: the model trained for next token prediction does not merely store features and add them. It amplifies some residual-stream directions and suppresses others until a small set of hypotheses controls the probability distribution that the model predicts. The transformer architecture, introduced by Vaswani et al. in 2017, is now widely reused in natural language processing and in other residual-stream stacks such as vision transformers. Decoder only models (GPT-style stacks, the LLaMA families, Mistral, Qwen) and encoder models (BERT, RoBERTa, DeBERTa) share the same core structure and differ mainly in masking and objectives. The residual stream is the central data structure. At each token position, a single high-dimensional vector accumulates additive updates from attention heads and MLP blocks across all the layers. The input vector starts as a linear combination of a token embedding and a positional encoding. Each subsequent layer reads from this stream, computes its contribution, and writes back additively. By the final layer, the residual stream carries the accumulated result of every previous layer computation. For a decoder only transformer, that final residual stream passes through a weight matrix (the unembedding) and a softmax to produce a probability distribution over the vocabulary. The loss function penalizes low probability on the actual next token. That objective, applied at scale, is a major reason language models acquire structure that later layers can select among. When interpretability talks about “what the model used,” it should mean which of those competing directions were allowed to write, and which later reads treated as live hypotheses, not which activations happened to be large.

Two architectural pieces create implicit competition among residual directions, and they should be named separately because they answer different selection questions. First, the self attention mechanism forces a probability distribution over context tokens via softmax on scaled query-key scores, then forms a weighted sum of value vectors. That is competitive allocation over positions, not a heat-map convenience. Query-key (QK) circuits decide, as a function of the current residual stream, which keys are eligible; output-value (OV) circuits decide what gets written if a key is selected. Second, saturating nonlinearities (GELU, SwiGLU, ReLU) create threshold-like behavior: a residual direction must be strong enough to pass the activation function or the MLP writes nothing along that path. Layer-norm then rescales across directions, so a direction that grows relative to others gains disproportionate downstream influence. From a Jacobian-sensitivity viewpoint, selection shows up as a low-dimensional set of residual directions with large derivatives into logits, while many other directions are effectively ignored for the current next token. Superposition makes component-level readings fragile, because many feature directions overlap in the same activation space. Empirically, circuit-level work on induction heads and task-specific circuits exhibits a common pattern: only a subset of residual directions dominate behavioral outcomes for a given prompt. That is what “emergent” adds to “selection.” None of these gates is hard-coded as a symbolic multiplexer. They arise from training dynamics and from the interaction of attention softmax with the activation function and the gradient landscape of next token prediction. A direction corresponding to a candidate answer may be present early, still contested in middle layers, and only later dominant. The transition from present-but-weak to dominant-and-causally-active is the selection event interpretability should measure. Neural networks of this family are especially amenable to that measurement because updates are additive and the residual stream is shared, so circuit analysis and patching can in principle trace which writes later layers chose to read. That reading is not cost-free. Treating every large activation as a selected hypothesis overstates the case: many large residual directions are unused for the current next token, and Jacobian-aligned analysis can itself over-weight directions that are sensitive but are not the program the model is running.

Attn(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V
# Q, K, V: query, key, and value projections of the residual stream
# softmax: maps scores to a probability distribution over positions
# V: what is written into the residual stream if a position is selected

Layer classes refine the same definition rather than replacing it. Research from 2020 onward has established that transformer layers specialize into early, middle, and late regimes, and that this is not simply a smooth gradient of abstraction. Early layers, roughly the first third of depth, select basic lexical and positional structure. They encode token identity and local syntactic patterns, including the consequences of positional encoding. In probing experiments these layers predict surface features well but often carry little information about the eventual answer. Middle layers carry more abstract or workspace-like content. Middle layers in transformers often share a more transferable representation space than early or late bands: skipping a block of middle layers can degrade benchmarks while leaving fluent text; reversing middle layers is sometimes tolerated. Those observations are consistent with middle layers refining a shared board rather than executing a rigid pipeline. They are not a license to treat middle layers as interchangeable. Replacing diverse middle layers with copies of a single layer typically fails, which indicates that each of the middle layers applies a different selection pressure inside a related representation space. Late layers align more tightly with next token logits. Crystallization of the answer into the top of the output distribution can continue deep into the stack, which means selection or refinement of hypotheses is not finished when middle layers have done their work. The practical implication is diagnostic, not mystical. Probing early layers tells you about feature encoding and positional encoding. Probing middle layers reveals workspace dynamics and which hypotheses are still in play. Probing late layers reveals output selection. Different tools are appropriate in each band, and localizing a behavior to a layer is not the same as naming the selected feature. Across all the layers, the residual stream remains the object later computation must choose to read. Vision transformers that reuse the same block structure implement analogous competition over patch tokens rather than words, which is why the same vocabulary of QK selection and OV writes is useful outside natural language processing.

A second, Jacobian-centered reading of the residual stream treats it as a competition arena rather than a passive accumulator. The Jacobian of logits with respect to the residual stream at a layer is a matrix whose high-sensitivity directions have causal influence on the vocabulary, while directions orthogonal to them are effectively invisible to the current next token even if their activation norm is large. Collections of those high-sensitivity directions give a low-dimensional map of what the model is currently selecting for at each position. That map is not the whole residual stream. Superposition implies that many unused or locally used features can sit in the same vector; selection is the process by which a subset of those superposed directions becomes causally active via softmax and the activation function. This competition picture is structurally close to global workspace theories in cognitive science: only a small subspace becomes broadly available to later reads, while other activity remains local. The analogy is structural, not metaphysical, and it does not settle claims about awareness. It does, however, change what “understanding a layer” requires. Understanding a layer requires saying which hypotheses were on the board and which directions still controlled the probability distribution after later cleanup. Practical experiments follow directly. Compute per-layer sensitivities on standard suites. Track how concentrated those sensitivities become through middle layers. Decompose residual-stream updates from individual heads and MLPs into components aligned with high-sensitivity directions, revealing which contributors were actually selected versus carried along. Compare decoder only models with encoder stacks and with vision transformers on the same geometry, because computer vision and multimodal fusion both reuse the residual-stream board. The residual stream after a previous layer is the menu; middle layers rewrite the menu; late layers cash it out as a next token. Interpretability that ignores that sequence will keep cataloging features that were present and never selected.

Why Attention Weights Are an Incomplete Explanation

The self attention mechanism implements selection over tokens in the input sequence, and that fact is the source of both the usefulness and the misuse of attention maps. For each query position, softmax over query-key dot products produces a probability distribution, often sharply peaked, that determines which context tokens contribute a weighted sum of value vectors. This is not passive feature detection. Value vectors gate which residual directions are imported into the current position. A head that attends to a representation of a candidate entity is proposing that hypothesis for further processing. A head that attends to a begin-of-sequence or sink token can implement a reset, reducing the influence of content tokens. Specialized heads documented in circuit papers — induction heads, name-mover heads, inhibitory heads — are therefore better described as selection operators than as feature detectors. They implement different verbs: copy, point, or suppress. Early-layer heads often mix or reset; middle layers more often retrieve or compare. None of that makes raw attention weights a complete explanation of the final output. A head can assign large attention weights to a token whose value vector is orthogonal to the directions that later control logits. Multi-head interference can cancel writes. Residual connections let information from a previous layer persist even if no head at the current layer attends to it. A downstream MLP can suppress what attention imported or amplify what attention down-weighted. Residual mixing and MLP override are why attention-is-not-explanation results should constrain practice rather than serve as a dunk. Jain and Wallace (2019) and follow-on work showed that learned attention weights frequently disagree with gradient and leave-one-out importance. Typical explainers still treat plotted attention weights as the selection story and never require a causal selection hypothesis. That is the gap this section is for. Attention maps are a hypothesis about where a head proposed to read. They are not, by themselves, a measurement of what was selected into the causally active residual stream. The counterargument is that discarding maps entirely would throw away the only cheap routing signal most teams can compute. The practical tradeoff is to keep maps as proposals and refuse to treat them as mechanisms.

Copy heads, pointer heads, and inhibitory heads make the incompleteness concrete because they implement different selection verbs on the same residual stream. A copy head that implements prefix-match-then-copy can show a crisp diagonal or offset pattern on an attention map, and in that special case the map often does track causal influence — but only after circuit work has shown that the OV write is actually a copy into a direction later layers read. A pointer head may attend to a name or slot in the input sequence while the information that matters is a pointer in the residual stream, not the surface token. An inhibitory head may attend broadly or to a duplicate token in order to write a suppression signal; the large attention weights then mark what is being down-selected, not what will appear as the next token. Name-mover heads in indirect-object circuits attend to a name and write it, but only after other heads have already selected which name is eligible. In each case the same style of heat map can correspond to retrieve, copy, or suppress. Geometric analyses of value space — high separability between selected and non-selected tokens versus collapse onto a sink attractor — are more informative than the map alone, because they describe what the OV circuit can write if the QK circuit selects. For visualization, raw attention matrices are therefore insufficient. A more informative display overlays attention weights with the projection of each head’s output onto logit-relevant residual directions. That overlay shows both where the self attention mechanism was directed and what, if anything, was selected for downstream causal influence. Practitioners working in natural language processing or computer vision should treat disagreement between attention weights and those projections as a failed selection hypothesis, not as a plotting error. The same caution applies to attention rollout and gallery-style maps on vision transformers: they inherit the assumption that routing weights equal selection importance, and residual-stream neural networks systematically violate that assumption.

A selection-aware attention analysis follows directly and should replace untested map reading in interpretability reviews. For each head at a layer, form the output (attention weights times value vectors), project that output onto high-sensitivity residual-stream directions at that layer, and treat the magnitude of the projection as the head’s contribution to selection. Heads with large attention weights and near-zero projection are proposing a read that later computation did not use. Heads with modest attention weights and large projection may be the actual selectors. This reframes attention patterns as candidate selection proposals that middle layers and late layers can accept or reject. It also explains why generic XAI lists that present SHAP, LIME, and attention rollout as sufficient for transformers systematically miss the mechanism. Those tools can highlight input tokens that correlate with the output while the model’s actual selector is a residual direction assembled across all the layers, including positional encoding channels and previous-layer writes that never appear as a single highlighted token. Compliance-oriented write-ups that stop at a token saliency map can therefore name the wrong selector even when they look precise. The recommended pairing is mechanical: every map ships with a stated hypothesis (this head selects that position in order to write this feature) and a causal test such as activation patching or path patching of the OV write. When the test fails, the map is retained as a routing clue, not as an explanation of why the model predicts a particular next token. That discipline is the difference between using the attention mechanism as evidence and using it as decoration. It applies equally to decoder only models and to vision transformers whose “tokens” are patches. In all of those neural networks, attention weights remain a first diagnostic of the input sequence, and they remain an incomplete account of selection.

MLP Blocks, Superposition, and Dictionary Learning Across Middle Layers

Attention is not the only selector. A second selector decides what is allowed into the residual stream as a feature, and therefore what later heads in middle layers can read. MLP blocks are per-token nonlinear operators. They read the current residual stream and selectively activate subspaces that correspond to intermediate concepts or hypotheses. Unlike attention heads, which route information between positions in the input sequence, MLPs transform information at a single position. Features in these neural networks are often polysemantic and context-dependent: a single unit may participate in several unrelated features depending on the surrounding residual stream. Despite that polysemanticity, many MLP directions are specialized to facts or task triggers, and only a minority of units are strongly active for a given token, which is already a form of sparse selection. The activation function is the gate. GELU and SwiGLU both produce threshold-like behavior. A residual direction must exceed a magnitude set by the activation function; below that threshold the neuron contributes nothing. Layer-norm before or after the block changes the menu by rescaling competitors. Together, MLPs and layer-norm are feature extractors that change what middle layers can select on subsequent reads. Composed programs make the second selector visible. Skip-trigrams and copy-suppression are not single-head gadgets. They are read/write programs in which an earlier write (often in early or middle layers) puts a feature on the residual stream and a later read decides whether to copy or suppress it. Localizing a behavior to a layer is therefore not the same as naming the selected feature. Saying “the behavior lives in layer 17” does not say whether middle layers selected a duplicate-token feature or a name-mover write. For next token prediction, MLPs near selected directions often act almost linearly onto logits through the unembedding weight matrix, which is why an isolated, correctly named MLP write can look transparent after the fact. The transparency is conditional on having named the thing that was selected. Recommended practice is to probe MLP activations across all the layers, decompose the MLP weight matrix into directions aligned with known circuits, and measure alignment of active MLP outputs with high-sensitivity residual directions. That is a different job from listing neurons that fire. It is the job middle layers force on anyone who wants to say what hypothesis the residual stream currently holds. The same second selector appears, with different surface statistics, in encoder models used for sentiment analysis and in vision transformers, because those neural networks also use an MLP to rewrite a residual stream after an attention mix. Computer vision fusion layers that borrow decoder only transformer blocks inherit the same menu-changing role for their middle layers. Dictionary-learning tutorials that catalog SAE features without asking which of those features middle layers wrote or read on a given pass therefore skip the selection question even when they recover a cleaner basis.

Superposition is why the unit of selection is so often a feature rather than a neuron, and why middle layers are the hardest band to read with unit-level tools. Elhage et al. (2022), in Toy Models of Superposition, showed that features can outnumber neurons, so one unit participates in many selectors and one selector is spread across many units. Overcomplete bases mean that middle layers can pack more hypotheses onto the residual stream than there are dimensions, then rely on sparsity and later competition to pick a subset. That packing is useful for capacity and hostile to naive interpretability — a genuine tradeoff, not a temporary inconvenience. If a neuron in the middle layers fires, you still do not know which of its superposed features was selected for the current input tokens. Sparse autoencoders and transcoders attempt to recover a candidate dictionary in which directions are closer to single features. That recovery is genuine progress, documented in the open Transformer Circuits program alongside residual-stream algebra and induction heads. It does not automatically answer the selection question. A dictionary can list features that were present in middle layers without saying which feature was written and which was suppressed. Reconstruction error and interference are not mere training nuisances; they are evidence that the thing being selected has not been isolated. There is a further tradeoff inside the dictionary itself. A coarser dictionary is easier to name and patch, but it bundles distinct selectors. A finer dictionary isolates more features, but reconstruction error and feature splitting make it harder to know whether the latent you patched is the object the layer actually selected. Feature absorption creates the same ambiguity: the latent you named in middle layers may be a fragment of the selector. Transcoders, which approximate an MLP map in a sparser basis, are closer to the second selector than input-only SAEs, because they try to explain the write, not only the residual stream before the write. Crosscoders that tie features across adjacent middle layers are closer still to a multi-layer selection claim, because they ask which feature identity persisted while middle layers transformed it. Even then, the right downstream test is causal: ablate or patch the latent in middle layers and ask whether the next token, or the downstream circuit, moves as the selection hypothesis predicts. Gallery-style SAE tours that never make that claim are disconnected from decision-level interpretability, however striking the auto-interpretability captions. The same caution applies when the dictionary is ported to vision transformers: a recovered texture or object latent in middle layers is a candidate for what was selected, not a demonstration that the residual stream used it. Across language models, the practical rule is stable. Use dictionary learning to propose the object of selection in middle layers. Use the residual stream and a later causal check to test whether middle layers actually selected it. Do not let a clean dictionary replace the selection hypothesis. Middle layers remain a workspace-like band in this picture: they are where many language models hold competing hypotheses, where superposition is densest, and where a misnamed feature most easily becomes a false explanation of the next token. Repeating the analysis on adjacent middle layers, not a single slice, is how you tell a transient write from a selected program. That is also why reports that only publish early-layer probes or last-layer logit lens leave the actual selection story in middle layers unexamined.

Induction Heads, In Context Learning, and Next Token Copy Circuits

Some circuits are selection all the way down, and those case studies set the standard for what it takes to claim you have found a selector. Olsson et al. (2022) documented induction heads as an emergent prefix-match-and-copy circuit that appears around a training phase change and implements in-context token selection: a previous token is matched, then the successor is written so the model can emit the matching next token. That is a two-step selection verb, not a static feature. Indirect object identification, isolated by Wang et al. in Interpretability in the Wild, decomposes another next token decision into duplicate-token heads, S-inhibition heads, and name-mover heads whose causal roles are not visible from any single attention map. Copy-suppression and skip-trigram circuits add further verbs: do not copy this token; fire if token A was followed by B at a distance. In context learning uses the same machinery at one more level of abstraction. Early layers encode the raw examples in the input sequence, including positional encoding that distinguishes the example slots; middle layers select a hypothesized task or rule from several that are representable; later layers apply the selected rule so the model predicts the next token. Dynamic routing means the same weights implement different functions when the prompt changes, which is why training-time analysis of a decoder only transformer cannot by itself certify which strategy will be selected at inference. Claiming you have found a selector, in any of these cases, requires more than a suggestive map in middle layers. It requires a named verb (match-then-copy, inhibit-then-move), a stated residual-stream object, and a causal intervention that moves the next token when that object is patched. Anything less is a correlation in neural networks that are very good at looking organized while doing something else. The same bar applies if the copy is over patches in vision transformers or over trajectory tokens in reinforcement learning: the circuit is a selector only after the write, not the plot, has been shown to control the final output. These case studies are also the clearest illustration of the article’s title claim. The implication is not that every behavior will decompose as cleanly as induction or IOI. The implication is that interpretability practices should treat “we found a selector” as a process-level claim — a verb plus an object plus a failed alternative — rather than as a plot that resembles a known circuit.

How Machine Learning Interpretability Practice Should Change

If transformer layers implement emergent selection processes, interpretability practices have to change at the level of hypotheses, not only at the level of tools. The first change is procedural: state an explicit selection hypothesis before running attribution or rollout. The hypothesis should name the layer band (early encoding, middle-layer workspace, or late output selection), the verb (copy, inhibit, or retrieve), the object (a residual direction or a head’s OV write), and the decision (which next token or which output class). Without that sentence, saliency scores and attention weights have nothing to be faithful to. The second change is causal pairing. Every map ships with a check: activation patching or path patching, with ablation as a coarser fallback. Activation patching asks whether replacing a component’s activation with its value from a counterfactual prompt flips the residual stream enough to change the next token. Path patching asks whether a specific path — this head’s write, read by that later head in middle layers — is necessary, rather than the component’s entire activation. Ablation asks what happens if the candidate selector is removed. Path patching is more precise than whole-component ablation, but it is also more expensive to specify and easier to miss an alternative path. Teams with limited compute may start with ablation and escalate only when the coarse test is ambiguous — a weaker causal claim, and an honest one. Naive perturbation wastes budget on non-selected directions. Selection-aligned patching constrains interventions to hypothesized directions and compares them with random directions of similar magnitude. If only the hypothesized directions move logits, the selection structure is supported; if random directions move them equally, the named selector is not isolated. Causal abstraction should treat abstract nodes as selected subspaces or hypotheses (“the model has selected name A as the mover”) rather than as single neurons. That shift is the core implication for teams that already run circuit analysis on language models and still write reports as if a highlighted token were the explanation. It also blocks a common failure mode in machine learning reviews: listing SHAP, LIME, and an SAE gallery as if any one of them answered the selection question. They do not, unless they are attached to a hypothesis that middle layers or late layers can be shown to have used.

Dictionary methods change practice in a narrower but important way: treat SAE latents as candidates for what was selected, not as the explanation. Teams should log, for each claimed feature, the reconstruction error, the interference with nearby latents, and the result of a patch in middle layers. If the patch does not move the next token, the latent was present and not selected. If several latents must be patched together, the dictionary has not isolated the selector. Transcoders should be preferred when the question is “what did this MLP write?”; residual SAEs when the question is “what was on the board?”; crosscoders when the question is “did the same feature persist across middle layers?” None of those tools replaces path patching when the question is “which path implemented the verb?” A lightweight decision tree is enough for most reviews. If the claim is about a route through named heads, path-patch. If the claim is about a feature basis in middle layers, recover a dictionary and patch the latent. If the claim is about what currently controls the probability distribution, probe the residual stream and report sensitivity concentration. If the claim is only that an input token mattered, do not stop at SHAP or LIME: check whether the model’s actual selector is that token or a residual feature assembled from positional encoding and earlier writes. Mixture-of-experts routing should sit on the same tree rather than in a separate literature. MoE routers are the explicit cousin of implicit dense-layer selection: both are learned competitive allocation, one with a discrete expert choice and one with softmax and saturating nonlinearities. Comparing router entropy in MoE language models with sensitivity concentration in dense decoder only models is a selection measurement, not a separate branding exercise. Instruction-tuned and reinforcement learning from human feedback variants of the same decoder only transformer can change which expert or which residual direction is selected without changing the fact that a selector exists. Practice that ignores MoE when it studies dense middle layers, or ignores dense middle layers when it studies routers, will keep missing the shared mechanism. Computer vision stacks that add expert routers on top of vision transformers should be read the same way: routing weights are attention’s cousin, and they still need a causal check before they explain a class decision.

Concrete workflow changes follow for interpretability teams that want selection claims to survive review. Write the selection hypothesis in the experiment ticket before any plot is generated. Record layer band, verb, object, and decision. Require a causal check on the same ticket; treat rollout and raw attention weights as pre-registered clues. When using dictionary learning, store latents as candidates with reconstruction metrics, and do not ship auto-interpretability text as a cause. Report negative results: hypothesized selectors in middle layers that did not patch. Report distributions across prompts, not a single clean example, because selection is prompt-dependent and single-prompt theater is a documented failure mode. Confirm with at least two methods (for example Jacobian-aligned attribution plus path patching). Include layer-norm in the instrumentation: run analyses on the post-norm residual stream, because a direction that looks large before norm can be suppressed after it. Distinguish activation magnitude from causal selection in every notebook template, because that confusion is the most common error in activation-based work on neural networks. Add task-conditioned notes. Arithmetic and long-context narrative need not share a commitment profile; middle layers may stay contested longer on multi-step tasks. When comparing different models — a base decoder only transformer versus an instruction-tuned sibling, or a language model versus vision transformers — compare selection structure (which verbs, which band of middle layers, how sharp the residual competition is) rather than only accuracy. Publish those profiles beside conventional scores. Researchers introducing new architectures should do the same. A model that reaches a benchmark with diffuse selection everywhere may behave differently under distribution shift than one that concentrates early. Finally, keep the residual stream, not the neuron, as the default object in write-ups. Ask what middle layers selected into that stream, what later layers read, and what the model predicts as a result. That is a smaller change in tooling than in standards, and it is the change that makes the rest of the stack — attention maps and SAE galleries — usable rather than misleading. Teams that already work across natural language processing and computer vision can apply the same ticket format to both, because the selection verbs transfer even when the input tokens are patches and the output classes are labels rather than a next token.

Evaluation, Governance, and Open Problems in Transformer Layers

Evaluation should target selection claims rather than generic saliency scores. Faithfulness tests that ask only whether a highlight correlates with the output will pass methods that name the wrong selector. A selection-faithfulness test asks whether intervening on the named object — a head path, a residual direction, or a dictionary latent — changes the next token (or the output class) in the predicted way, and whether matched interventions on non-selected objects do not. Useful accompanying metrics are qualitative even without new unpublished numbers: concentration of sensitivity into few residual directions; depth at which counterfactual residual swaps stop flipping the prediction; stability of selected directions across paraphrase; agreement or disagreement between attention weights and OV-to-sensitivity projections. These should be computed on public suites and on more than one family of language models, including encoder models used for sentiment analysis and decoder only models used for open-ended generation. Visualization, if used, should restate those measurements: layer-wise concentration, trajectories of the selected subspace through middle layers, and attention overlays colored by causal alignment rather than by raw attention weights. Cherry-picked clean plots remain a pitfall. So does generalizing a selector found in one decoder only transformer to all neural networks, or from language to computer vision, without repeating the causal check. Mixture-of-experts routing belongs in the same evaluation sheet. Router choices are explicit selection; if a report explains a dense model with a token map and an MoE model with a routing table, and never asks whether both are learned competitive allocation over a residual stream, the evaluation is incomplete. Safety evaluations should monitor whether selected directions change under red-team prompts, because a model can keep fixed weights while middle layers select a different strategy. Steering vectors succeed or fail depending on whether they land in currently selected residual directions. None of that requires claiming that full mechanistic interpretability is solved. It requires refusing to treat untested maps as evaluations of selection. The implication for evaluation practices is narrow: score the selection process you named, not the prettiest saliency overlay.

Governance pressure makes the methodological point operational. Regulation (EU) 2024/1689 (AI Act) imposes transparency and information duties so deployers and affected persons can interpret system output, with heightened duties for high-risk systems. The NIST AI Risk Management Framework treats Explainable and Interpretable as a distinct trustworthy-AI characteristic and separates mechanism-level explanation from output-level interpretability in context of use. The OECD AI Principles call for transparency and responsible disclosure so people can understand AI outcomes and challenge them. Those texts are not circuit papers, but they are not satisfied by a convenient surrogate. If the explanation names input tokens that a SHAP or LIME map highlights, while the model’s actual selector is a residual feature assembled in middle layers, the disclosure can be fluent and wrong. That is a major reason mechanistic evidence belongs in compliance packets for high-risk language models, not only in research blogs. The institutional tradeoff is familiar: a fully mechanistic account can be accurate and unusable to the affected person, while a simplified token map can be usable and wrong. Neither pole satisfies NISTIR 8312 on its own. An explanation can be meaningful to a non-technical audience and still have to declare its knowledge limits: this map is a routing proposal; this SAE latent is a candidate. NISTIR 8312’s demand for measurable explanation accuracy is closer to selection-faithfulness than to aesthetic clarity. Deployers who must describe why a decoder only transformer produced a next token, or why an encoder produced one of several output classes in sentiment analysis, should be able to say which selector was tested. The same standard applies when the system is a vision transformer in computer vision or a policy network in reinforcement learning that reuses transformer layers. In each setting, the legal and institutional demand is that people be able to interpret and challenge outputs. That demand is only met if the named mechanism is the one the residual stream actually used. Treating generic XAI method lists as sufficient is therefore not just a scientific mistake; it is a governance mismatch. Conversely, mechanistic reports that never state an audience or a knowledge limit fail the same institutional tests from the other side. Selection-aware interpretability is how those two literatures meet without either inventing a completed science or pretending a heat map is a mechanism.

Open problems are specific, and they should be stated without hype in either direction. Architectural questions include whether explicit selection modules or regularizers can make the selected subspace reportable by construction, and whether training objectives can penalize hidden hypotheses that remain off-board until an adversarial prompt selects them. Scaling questions include how selection sharpness and the role of middle layers change with width and depth, and whether instruction tuning or reinforcement learning from human feedback qualitatively retunes selectors or only recenters them. Multi-token concepts remain awkward for tools whose natural basis is a single next token. Cross-domain transfer is under-studied: vision transformers and multimodal fusion layers all have middle layers and a residual stream, but most selection case studies are still text-only. Mixture-of-experts routing should be compared directly with dense softmax selection instead of living in a separate literature. Neuroscience analogies — competition, winner-take-all, global workspace — are useful for generating hypotheses about middle layers and misleading if they are used to settle consciousness claims. This article remains neutral on those claims. Workspace-like selection in transformer layers is a computational observation about bandwidth and competition; it is not evidence of subjective experience, and it is not a proof that such experience is absent. Limits of current methods should be listed beside the open questions. Attention weights omit residual mixing. Dictionary learning can recover a basis and still miss the selector. Patching is causal only for the interventions you ran. Jacobian-aligned analysis names sensitive directions and still needs a verb. Positional encoding channels are easy to ignore and often part of the selector. Different models, including different decoder only models, need not share a commitment profile. Results from one stack should not be treated as universal properties of all the layers in every transformer. Future work that publishes selection profiles, negative circuit findings, and governance-facing knowledge limits will do more for interpretability practices than either a claim of imminent completeness or a claim of permanent impossibility. Readers who want to apply the view can start by writing a one-sentence selection hypothesis, computing a causal check on an open decoder only transformer, and refusing to ship an attention map or SAE feature as an explanation of a next token until that check is done. The residual stream will still be a competition arena. Middle layers will still rewrite the menu. The job is to name what was selected, and to say how you know.

Related reading: LLM misalignment detection with J-Lens.

Editorial Team

Editorial Team

The J-Space Editorial Team curates and synthesizes research in mechanistic interpretability and AI alignment. With a commitment to precision and empirical rigor, the team distills complex technical developments into accessible yet authoritative analyses for researchers and engineers working on interpretability and safety.