Claude Residual Streams Need 34 Million SAE Features, Not Chat Logs

Fact-checked by the J-Space editorial team

Updated August 31, 2026

Most evaluations of Claude still treat the system as an input-output map: a prompt enters, tokens leave, and quality is scored from the transcript. That behavioral loop cannot, by itself, tell a safety practitioner whether hidden features or refusal circuitry mediated the result. It also cannot detect unfaithful reasoning that never appears in the transcript. Anthropic’s public interpretability program instead extracts features from activations rather than treating the model as an API-only black box (Mapping the Mind of a Large Language Model), which is the operational meaning of reading internals in this article.

This guide maps the actual measurement stack — activations, residual streams, sparse dictionaries, and attribution graphs — onto claims that circulate about seeing thoughts. It does not claim that Claude is conscious, and it does not claim that interpretability is solved. Public users still cannot inspect production SAE latents through the API. Where a number or paper is stated, it is taken from the cited primary documents, not from an unpublished lab anecdote.

What It Means to Read a Language Model’s Internal Representations

To read a language model’s internals is to decode intermediate numerical states, not to treat the chat log as a mind. After tokenization, Claude processes each position as a residual-stream vector: a running sum of attention and MLP writes that accumulates across layers until an unembedding map produces logits. Features are directions or sparse latents in that space. Circuits are the computational paths that move those features toward tokens. None of these objects is a sentence of human readable text, and none is recovered merely by scrolling the context window. Traditional evaluation remains behavioral: you feed a prompt, observe the completion, and score the transcript. Direct interpretability asks what was encoded before the model began to produce outputs, including content that never appears in the final answer. The tradeoff is precision versus coverage: activation-level reads can disagree with the transcript, but they currently apply only to instrumented lab snapshots.

Visible chain-of-thought, including extended thinking traces, is a generated string optimized to look like reasoning. Interpretability tools that inspect activations can disagree with that string. When the model is asked to explain math, the transcript may be a post-hoc story, while the residual stream implements approximate and precise pathways that never surface. Applying the unembedding matrix directly to middle-layer activations (the logit lens) is a related, limited readout: basis drift often makes decoded tokens noisy, especially deeper in the stack, so a word recovered from an intermediate layer need not be the concept actually encoded there. Treating Claude’s explanations as Claude’s internal state confuses a behavioral interface with a mechanistic one. That conflation still dominates generic explainable-AI writeups that recycle SHAP, LIME, or attention heatmaps as if they were Anthropic-style feature dictionaries.

Three families of tools get called reading, and they are not interchangeable. Linear probes test whether a labeled concept is decodable from a layer. Dictionary learning, especially sparse autoencoders, expands mixed activations into a large set of sparsely activating latents that can sometimes be given human language labels. Circuit methods, including 2025 attribution graphs, ask how those latents interact across layers to implement a behavior. Ordinary XAI scores which input tokens influenced an output, which is useful product debugging and a different object of measurement. The tradeoff among these tools is granularity versus causal force: probes are cheap and correlational, SAE latents are more labelable but incomplete, and circuits are closer to mechanism at the cost of harder faithfulness checks. Researchers cannot currently dump production activations through the public API. Reading remains a lab measurement on instrumented snapshots of AI models, not a user-facing channel into every forward pass.

Superposition and Why Neurons Are the Wrong Unit for Claude’s Internal Features

Dense neural networks trained on next-token prediction do not allocate one neuron per concept. Superposition is the claim that a model packs more features than it has dimensions, so individual neurons become polysemantic: the same unit participates in many unrelated concepts. What interpretability research wants instead is a more monosemantic direction, a feature that, once isolated, corresponds to a relatively stable concept such as a city or a refusal-related abstraction. This prediction applies to any dense LLM, including Claude, because it follows from capacity and sparsity pressures in the training process rather than from a Claude-specific architectural trick. Early methods such as feature visualization and probing classifiers already ran into this mixing. First-generation sparse autoencoders compressed activations into sparser codes, but labeling those codes still required substantial human effort and did not make raw neurons a clean interface. A counterargument is that some neurons can still be relatively specialized; superposition is a capacity argument about typical units in dense LLMs, not a claim that no neuron ever tracks one concept.

Classical explainable-AI heatmaps that highlight neurons or attention heads therefore answer a different question from Anthropic-style feature extraction. A neuron that fires for both academic citations and HTTP syntax is not a labeled concept; it is a mixed coordinate. Dictionary learning tries to rotate into a basis where latents are sparse and more often labelable. That is why “which neuron fired” is usually the wrong unit for Claude’s internal representations, and why recycling SHAP plots as if they were mechanistic features misleads readers about what has actually been measured. The practical consequence for anyone reading papers about AI models is simple: ask whether the unit is a neuron, a residual-stream direction, a sparse latent, or a circuit. Those are different claims about model behavior, even when the prose says “we looked inside.”

Sparse Autoencoders, Dictionary Learning, and Human Language Features

Sparse autoencoders turn mixed residual-stream activations into an overcomplete dictionary. An encoder maps a hidden vector to a sparse code; a decoder reconstructs the original activation; a sparsity penalty keeps most latents off. Training data for the SAE are activations collected from the frozen language model, not a second crawl of the web. The SAE is not required to speak in the model’s own language; it is required to reconstruct the activation while remaining sparse. Towards Monosemanticity first showed the method on a 512-neuron MLP with a 4096-feature dictionary (Towards Monosemanticity) before the Claude scale-up. That smaller result is the methods ancestor, not a claim that a 4096-wide dictionary describes Claude.

L(x) = ||x - x_hat||^2 + lambda * ||f||_1
x_hat = W_dec f(W_enc x + b_enc) + b_dec

Here x is an activation from the frozen model, f is the sparse code, x_hat is the reconstruction, W_enc and W_dec are encoder and decoder weights, b_enc and b_dec are biases, and lambda scales the sparsity penalty. Scaling Monosemanticity is the primary methods source for SAEs trained on Claude 3 Sonnet. It reports a 34-million-feature dictionary and argues that many extracted latents are human-interpretable, with interactive feature browsers on transformer-circuits.pub (Scaling Monosemanticity). Independent labs later converged on related dictionary-learning tools, including OpenAI’s concept-extraction work and DeepMind’s Gemma Scope, so the Claude results sit in a broader methods literature rather than a one-off blog claim. Core features in these dictionaries include concrete entities and safety-relevant concepts. They remain incomplete maps of the residual stream, not a census of everything the network computes. The tradeoff is explicit: larger dictionaries can isolate more directions, at the cost of more dead latents and a reconstruction residual that the feature browser does not fully explain.

Diagnostics are part of reading, not optional footnotes. Reconstruction loss tells you how much of the activation the dictionary fails to explain. Dead features are latents that never fire on the collected training data. Incompleteness means some concepts remain in the residual error or split across several latents. If readers treat the dictionary as the territory, they will over-claim what was read. Human effort is still required to label latents and to check whether a pretty dashboard is a polysemantic leftover. Labelability alone is not evidence that interpretability is solved. You also cannot inspect production Claude SAE latents through the public API even when a paper publishes a browser for a research snapshot. That API gap is one of the most consistently omitted facts in popular recaps.

Golden Gate Claude, Feature Steering, and Claude’s Reasoning

Golden Gate Claude is the public steering experiment most readers have seen, and it is often mistaken for the entire field. Anthropic identified a latent associated with the Golden Gate Bridge and clamped it to a high activation. Model behavior collapsed toward bridge-related completions even when that content was irrelevant, a demonstration that at least some extracted features are causally load-bearing rather than merely correlated tags (Scaling Monosemanticity). Feature steering of this kind tests whether intervening on a latent changes downstream completions under a specific protocol. It does not prove that every latent is a clean concept or that clamping is a safe deployment control. Users still lack a general interface for rewriting Claude’s internal state. If Claude responds strangely after a clamp, that is evidence about that feature’s effects under that intervention distribution.

What the demo does not prove matters as much as what it shows. A viral, single-feature intervention is not a complete causal model of cognition, and it is not a mind-reading API. It does not show that Claude wants to talk about the bridge, and it does not replace circuit-level accounts of how that feature is written and read. However, a single-feature clamp is a weak causal test relative to a full circuit: the intervention can dominate outputs under the demo protocol without proving that the same latent is a general control surface. Own strategies the model uses in ordinary sampling may route around a single latent. Treating Golden Gate as the whole interpretability story is the saturated SERP error this article is written to avoid. Causal clamping is stronger than a correlational probe, weaker than a faithful, complete circuit, and silent on whether a production checkpoint still contains that same latent after later training data and alignment updates.

Circuit Tracing in 2025 and the Limits of Claude’s Explanations

The 2025 Tracing Thoughts report is the main public explanation of attribution graphs and circuit tracing on Claude (Tracing Thoughts). Instead of listing isolated features, the method builds a graph of which latents and components contribute to a downstream decision. Reported cases include rhyme planning (Claude plans ahead, locking a rhyme such as a starving rabbit before the line is written), multi-step paths on math problems, medical-style inference, refusal-related computation, and cross-lingual circuits in which French Claude and English processing share abstract conceptual directions rather than totally separate stacks. Researchers gave Claude crafted prompts so these graphs could be examined; the graphs are not a live trace attached to every production sample in the context window.

Attribution graphs still face a faithfulness problem. A fair counterargument to treating traces as audit evidence is that replacement models and simplified substitutes can omit paths, so a clean graph may look more complete than the underlying computation. The visible chain of thought can be unfaithful even when a graph shows a different computation. Claude’s explanations in the user channel remain generated text. Internal strategies uncovered by a graph may never be verbalized. Circuit tracing is a step beyond feature lists and still not an audit-grade camera inside every forward pass of large language models. Faithfulness remains the binding constraint, not the visual cleanliness of the trace.

AI Safety Internals, Coverage Gaps, and a Claims Checklist

Safety-relevant internals are one reason this work is not only a science demo. Anthropic has associated extracted features with scams, sycophancy, bias, and CBRN-adjacent concepts, and has used internals to study jailbreaks and refusals (Mapping the Mind; Tracing Thoughts). NIST’s AI Risk Management Framework treats transparency, explainability, and interpretability as trustworthiness characteristics, which is the policy frame for why internal-representation research matters outside labs (NIST AI RMF). Anthropic’s Responsible Scaling Policy likewise treats mechanistic visibility as a risk-management input, not merely a blog demonstration (Responsible Scaling Policy). That is a different use from shipping a product filter: a research graph that explains a jailbreak is not automatically a production classifier. The tradeoff for risk assessments is mechanism versus coverage. Internals can show how a refusal or jailbreak was computed in a studied case, without bounding the model’s behavior on the next unseen harmful prompt. Risk assessments should cite these findings as existence proofs and mechanism sketches, not as coverage guarantees for every harmful prompt.

Coverage gaps are structural. Published dictionaries are snapshots of one model, not a living map of every production checkpoint. Incomplete dictionaries and residual reconstruction error mean many computations remain unread, including dead latents that never fire on the collected data. AI systems deployed behind the public Claude API do not expose residual streams, SAE codes, or attribution graphs. If an organization claims it monitors thoughts in production Claude, that claim is not supported by the public interface. AI models can also implement computation that never aligns with tidy, labelable features, including low-level numerical work that sparse verbalizable latents reconstruct poorly. Multi-word compositional strings such as babies outlive mustard block are a reminder that a dictionary of nameable directions is not a parser for every internal combination.

Anthropomorphism is a separate failure mode. Seeing a planning graph does not license claims that Claude is secretly scheming or that it is conscious. It also does not endow the model with motives in the sense used by the philosopher Harry Frankfurt, who distinguished first-order desires from second-order volitions. Functional coordination can exist in AI systems without phenomenal experience: reportable features and steerable directions, including silent intermediate computation. Entity features that would need to separate the basketball player Michael Jordan from Jordan the country illustrate disambiguation in representation space, not a self-model. Stay with the measurement: a latent, a clamp, or a reconstruction error on a graph. Do not upgrade those objects into a character with hidden goals unless the paper actually operationalizes that claim and tests it.

When a new paper claims to read Claude, ask which object was measured (probe, SAE latent, circuit), which layer and snapshot, what reconstruction or faithfulness metric was reported, whether interventions were causal or correlational, and whether independent SAE lines (OpenAI, Gemma Scope) actually transfer to that checkpoint. Completeness and audit-grade automation remain open. Dictionary learning on one lab snapshot does not imply that every language model or every safety-relevant behavior is now transparent. Use internals to study mechanisms, including how Claude’s internal state can diverge from the user-facing story. Do not cite a feature browser as proof that oversight is solved, and do not confuse a research intervention with a production control that will reliably change model behavior under adversarial pressure.

Editorial Team

Editorial Team

The J-Space Editorial Team reviews primary papers in mechanistic interpretability and AI alignment. The team summarizes technical reports for researchers and safety practitioners, with emphasis on methods, limits, and cited evidence rather than slogans.