Superposition and Dictionary Learning Features

Anthropic’s toy-model report (published 14 September 2022) names superposition as the phenomenon in which models represent more features than they have dimensions when features are sparse. The 5 October 2023 dictionary-learning report then looks for units other than individual neurons: features that correspond to patterns, linear combinations, of neuron activations, because many model properties do not appear on single neurons. He et al. (arXiv:2402.12201v1, submitted 19 February 2024) supply an explicit sparse-coding objective and a small-transformer case, including where attribution fails. The aim here is to state the sparsity condition, the reconstruction objective, and the controls that separate a feature label from a causal claim.

Sparse features make superposition possible

The 14 September 2022 report defines the object of study before it defines the phenomenon. The models are small ReLU networks trained on synthetic data with sparse input features. The question is how and when those models represent more features than they have dimensions. The authors call that superposition. The page description repeats the same condition: small networks can represent more features than they have dimensions when features are sparse.

The mechanism they state is a tradeoff, not an unqualified gain. The abstract says: “When features are sparse, superposition allows compression beyond what a linear model would do, at the cost of ‘interference’ that requires nonlinear filtering.” Sparsity is the condition the abstract names for compression beyond a linear model. The stated cost is interference that requires nonlinear filtering. The models used to study that tradeoff are small ReLU networks trained on synthetic data with sparse input features.

Applied to a trained language model, the same idea is a hypothesis, not a measurement copied from those toy nets. He et al., citing Arora et al. (2018) and Elhage et al. (2022b), define the superposition hypothesis as the assumption that the model learns to represent more linear features than it has hidden dimensions. In Section 4.2, citing Elhage et al. (2022b), they treat that hypothesis as what makes hidden activations problematic to interpret, because the model is assumed to place more linear features than it has hidden dimensions into the activation space. They do not attach the vision-model comparison to that assumption. They attach it only to sparse features: “Additionally, the sparse nature of features in language models make it more challenging to find monosemantic neurons than in vision models” (Cammarata et al., 2020b; Goh et al., 2021, as cited there). That is their framing of prior work. It is not a new count of features in the Othello model.

Why a single neuron does not reveal the feature

If features are overlapping directions, a neuron is only a coordinate axis of the activation. A feature can have a component on many neurons and line up with none of them. A neuron can then fire for several features that all load on that axis. Inspecting one neuron’s highest activations will not list the features the dictionary later separates.

The 5 October 2023 report, “Towards Monosemanticity: Decomposing Language Models With Dictionary Learning,” states this as a result about units of analysis. The authors say there are better units than individual neurons, and that they built machinery to find those units in small transformer models. They define the units, called features, as corresponding to patterns, linear combinations, of neuron activations. In a transformer language model, they decompose a layer with 512 neurons into more than 4,000 features that separately represent things such as DNA sequences, legal language, HTTP requests, Hebrew text, and nutrition statements. The page says most of these model properties are invisible when looking at the activations of individual neurons in isolation.

That sentence is evidence against neuron-level reading for that layer, not a license to treat 4,000 as a universal feature count. The retrieved page does not name the checkpoint, which layer, the corpus, the dictionary width, or the sparsity penalty. “More than 4,000” is a lower bound stated for that layer. He et al. later cite the same paper, Bricken et al. (2023), Transformer Circuits Thread, for a different quantity: typically 10–20 sparse activations on a 512-dimensional layer. That figure, as they use it, is how many dictionary features fire, not how many features the dictionary contains. The two numbers answer different questions and come with different missing fields. This article does not remeasure either one.

What dictionary learning optimizes

He et al. (Section 2.1) state the search as sparse coding with an autoencoder. For an activation x at a chosen site, they look for overcomplete bases d such that

x ≈ Σ_i w_i d_i, subject to minimizing the number of positive coefficients w_i.

Here w_i is the activation magnitude of dictionary feature i, and d_i is a unit vector for its direction. The sparsity constraint is what they say forces the dictionary toward fundamental features implicit in the representation, rather than an arbitrary basis that also reconstructs x. Their Figure 1 captions the picture: an activation at a given position and input is decomposed into a sparse reconstruction of dictionary features. The encoder sets each feature’s magnitude, a ReLU gates it, and the decoder maps the sparse code back toward the activation.

What “found” means

The procedure returns directions and coefficients that reconstruct activations under a sparsity penalty. A further claim, that one direction is one human-readable concept, and that the model uses it, needs separate evidence. The October 2023 report offers that evidence as examples of features that do not show up on isolated neurons. He et al. offer a different kind: unsupervised recovery of feature types that earlier probing had already named in Othello-GPT. Neither statement, as printed in these sources, is a proof that every dictionary element is a causal variable.

The encoder and decoder are not the same object. The decoder direction is the feature’s vector in activation space. The encoder, with its bias, estimates how strongly that feature is present. He et al. train an untied encoder and decoder, so the two maps are not forced to be transposes, and they put an L1 penalty on the hidden code (Appendix A.2.2). Those are their training choices, not the only definition of dictionary learning. The extract does not print the numerical L1 coefficients, only that they used smaller coefficients in later layers and may have set them too small.

Which activation the dictionary is fit to

The objective does not say which tensor to decompose. He et al. (Section 2.2 and Figure 2) compare five sites. As they summarize prior work, dictionaries had been trained on word embeddings, on MLP hidden layers (Bricken et al., 2023), and on residual streams (Yun et al., 2021; Cunningham et al., 2023). They argue instead for the three writes into the residual stream: word representations, each attention output, and each MLP output. The extracted paper does not report a shared metric, reconstruction, a monosemanticity score, or circuit recovery, that ranks those five sites on one model. The table is their design commentary, not a measured winner.

The parameter range is a formula, not a timing experiment. They write the blow-up as (d_MLP / d_model)² and summarize it as about 4–16 times. For their Othello model only, Appendix A.2.1 gives d_MLP = 512 and d_model = 128, so 512 / 128 = 4 and 4² = 16, the top of that stated range. The assumption is theirs: counting trainable dictionary parameters if one decomposes the wider MLP hidden space rather than the model-width output. It is not a wall-clock result, and it does not apply to the 512-neuron layer on the October 2023 page, whose model width is not stated there.

Their architectural assumption is pre-norm. They say one only needs the components each module writes into the residual stream, then their sum. They also linearize LayerNorm by treating the input standard deviation as a constant rather than a function of the input (Appendix A.1), which leaves the numerical result of LayerNorm unchanged while making it linear in a decomposition. They note that the same trick is already used for direct logit attribution. If that approximation is poor, residual-stream attributions are poor even when the dictionary reconstruction is faithful.

What an Othello dictionary recovered

He et al. use Othello-GPT as a case, not as a stand-in for the Anthropic layer. Appendix A.2.1 says they retrained the model rather than using the released checkpoint from Li et al. (2023). Their network is a decoder-only transformer with about 1.2 million parameters, 6 layers, hidden size 128, and 512 neurons in each MLP. They trained on about 10⁹ sequences, removed dropout because they say dropout increases redundancy and makes circuits harder, and report legal-move accuracy of 99.5%, against 99.9% for the original 8-layer model. The extract does not state the evaluation-set size behind those two percentages. The task is legal-move prediction, not tactics. They denote the Y-th dictionary feature in layer X as LXAY for attention output and LXMY for MLP output.

Dictionary width and a training failure they report

Each dictionary is an autoencoder with 128-dimensional input and output and 1,024 hidden units, untied weights, and an L1 penalty (Appendix A.2.2). They reduced the L1 coefficient in later layers because activations grew, citing a trend in Dettmers et al. (2022); they do not print the coefficient values. Figure 10’s caption states the limitation of that run: near-perfect reconstruction error and too many dead neurons. In the appendix text they say about 75% of neurons in each dictionary were dead, and they treat too-small α as the likely cause. The companion panel defines an active feature as one activated at least once in 50,000 inputs. Per-layer active counts are drawn in the figure and are not printed as numbers in the extract, so they are not repeated here.

Sparsity in this model is also not the 10–20 regime they attribute to Bricken et al. Figure 13 reports average L0 over 10,000 tokens. In higher MLP layers that L0 sometimes approaches d_model = 128, so the decomposition is not very sparse. They attribute the contrast with Bricken et al.’s typically 10–20 active features on a 512-dimensional layer to a simpler, denser Othello world and to dictionary training they believe is suboptimal. Those are different models and different dictionary widths. The comparison is theirs; it is not a remeasurement of the October 2023 layer.

Feature labels are summaries of top activations

Section 3.2 is a reading of activating examples, not an ablation study. They report current-move features in L0 attention and MLP (for example L0A53 for the current move at g-3, and L0M996 for the current move at a-4 flipping the piece above), with similar features in L1M through L3M; board-state features from L1A through L4M (for example L1A626, f-1 is the opponent’s piece); empty-cell features in L5A (L5A336, c-3 is empty); and legal-move features in L5M (L5M733, b-3 is legal), which they say also contribute to the matching logit when active. They conclude that dictionary learning finds, unsupervised, all feature types previously found by probing. That conclusion is their interpretation of this survey.

Appendix A.3 is explicit that single board images invite interpretability illusions. For a feature they take the top k = 2,048 activating inputs out of n = 1.2 million board states and aggregate whose turn it is, where the move was played, which squares were legal, own versus opponent versus empty, flips, empty cells, and game length. L0A622 is labeled “current move = f-1” because, among those 2,048 inputs, every move was played at f-1 (Figure 11 and the interface in Figure 12). That is a correlation between activation and a board statistic. It does not show that the feature is necessary for the logit.

Attribution is not a causal patch

Section 3.3 says the circuit analysis is on certain inputs and that they claim no universal understanding of Othello-GPT. The OV, QK, and MLP walkthroughs are author readings of decompositions on chosen positions. For attention outputs they use the linear OV and QK structure, after the LayerNorm approximation above. For MLP features they do not have that linearity, so they define Approximate Direct Contribution (Definition 2.1). For a self-gated nonlinearity of the form x · σ(x), they freeze the gate at the full MLP input and treat one lower-level feature c_kd_k as the linear piece:

ADC(c_kd_k) = W_{encoder, Y} W_out ( (W_in c_kd_k) · σ(W_in x) ).

They say this qualitatively tracks features that increase neurons with a positive linear effect on the output feature, because for a monotonically non-decreasing non-negative σ the derivative of xσ(x) is non-decreasing. It misses suppression of a neuron that would have had a negative effect, whenever σ is driven to 0: the frozen gate then contributes 0, so ADC through that neuron is 0. They also report cases, including L4M319 in the text around Figure 9, where ADC assigns the activation mainly to decoder biases. They conjecture an OR-like neuron of the form c = 1 − ReLU(1 − a − b), and say that both ADC and patching theoretically fail to identify both inputs of that gate. QK decomposition has a separate gap: it splits one pre-softmax score into feature-pair terms and does not explain relative strengths after softmax, because changing one score moves the pattern over the whole sequence.

The one numeric comparison with patching is correspondence, not identity. In Section 5.1.3 they sample 10 input sequences, take the top 3 activated dictionary features in each of 6 MLP layers, and state that this is 180 features (10 × 3 × 6). For each, they take the top 5 lower-level features by ADC and by direct patching. The average intersection-over-union is reported as 0.68; the extract does not give the per-feature scores or the rounding rule. They call that relatively high correspondence. They do not claim their method dominates patching. Table 1 compares complexity only: causal patching is O(n) model forwards and is out-of-distribution; direct patching and their method are O(n) module forwards, with n the number of lower-level features; they say only the patch-based methods have the out-of-distribution problem. “Causal” here means ablating a low-level feature and rerunning later modules. “Direct” means linearizing the other low-level features without that rerun. Those are different interventions from reading top activations.

Limits that stay attached to the results

The toy-model claim supported by the retrieved 14 September 2022 page is the abstract’s definition: sparse synthetic features, small ReLU networks, compression past a linear model, interference removed by nonlinear filtering.

The decomposition of 512 neurons into more than 4,000 features is one layer in one transformer language model, as stated on the 5 October 2023 page. Checkpoint, layer index, and dictionary hyperparameters are unknown from that page. He et al.’s citation of typically 10–20 active features on a 512-dimensional layer is a different statistic, attributed to Bricken et al. (2023), and is not recomputed here.

The Othello evidence is one group’s retrain: 6 layers, hidden size 128, MLP width 512, no dropout, about 10⁹ sequences, dictionaries of width 1,024. This source bundle does not contain an independent replication of those dictionaries or circuits. Feature names summarize top activations (k = 2,048 of 1.2 million boards in their interface). Circuit graphs are input-specific, which the authors say explicitly. Agreement with direct patching is a mean IOU of 0.68 on the 180-feature, top-5 comparison above, not a match, and not a causal-patching result. Their own dictionary run had about 75% dead features and, in late MLPs, L0 sometimes near 128.

Linearized LayerNorm and ADC are approximations with stated blind spots: inhibitory paths when the gate is zero, OR-like neurons, and softmax competition across the sequence. Nothing in these sources is a claim about consciousness or sentience. The measurements are about activation geometry, reconstruction, and, in the Othello case, input-conditional attribution.

Your Action Plan

1. Write the sparsity condition into the claim. State superposition, as in the 14 September 2022 abstract, as representing more features than dimensions when features are sparse, with interference that requires nonlinear filtering. Do not import He et al.’s “more linear features than hidden dimensions” into that abstract. If the source does not give a sparsity threshold, do not invent one.
2. Do not read a neuron as the feature. Record whether the evidence is a neuron activation or a direction. In the 5 October 2023 report, features are linear combinations of neuron activations, and most of the listed properties were invisible on neurons in isolation.
3. Choose the site before training the dictionary. He et al. argue for every module that writes the residual stream, embedding, attention output, MLP output, under pre-norm. If you decompose MLP hidden states instead, account for (d_MLP / d_model)² and for directions in the down-projection nullspace. Treat that as their argument, not a measured ranking.
4. Report the sparsity–reconstruction tradeoff, not reconstruction alone. Their objective is a sparse positive combination of unit directions. Their Othello setup was 128 to 1,024 to 128, untied, with an L1 penalty and smaller penalties later in the network. Publish reconstruction error, dead-feature rate, and L0 on a stated token count. Near-perfect reconstruction with about 75% dead features was, in their appendix, evidence that the penalty was too small.
5. Label features from many top activations, then separate correlation from intervention. Use an aggregate over high-activating inputs, they used 2,048 of 1.2 million board states, rather than one salient example. A legal-move feature that co-occurs with a logit is not yet a circuit.
6. Name the operator and the input if you attribute a circuit. Linear OV or QK decomposition, ADC, direct patching, and causal patching answer different questions. On their sample, ADC and direct patching agreed at average IOU 0.68 for top-5 contributor sets, calculated over 10 sequences × 3 features × 6 MLP layers. Do not promote an input-specific graph to a general motif; they decline that claim.
7. Keep every width inside the experiment that measured it. The 512-neuron layer and the hidden-size-128 Othello dictionaries are not substitutes. If checkpoint, layer, penalty, evaluation-set size, or an independent replication is missing, say so instead of importing it from another model.

1. Sparse features make superposition possible
2. Why a single neuron does not reveal the feature
3. What dictionary learning optimizes
4. Which activation the dictionary is fit to
5. What an Othello dictionary recovered
6. Limits that stay attached to the results
7. Your Action Plan