In Gao et al., arXiv:2406.04093v1 (6 June 2024), Section 4.3, the interpretability illusion is a citation gloss of Bolukbasi et al., “An interpretability illusion for BERT” (arXiv:2104.07143, 2021), not a restatement of that paper’s checkpoint, datasets, or corpus contrast. Gao et al. use it for a specific failure: random activating examples can show a recognizable pattern that suggests a feature explanation, while the explanation is too broad, so recall is high and precision is poor. The Bills et al. (2023) case they give is a feature that activates at the end of “don’t stop” or “can’t stop,” whereas an explanation that activates on every “stop” still gets a high automated interpretability score because that score depends disproportionately on recall; Gao et al. do not print a numeric score or name the model for that example. They then build Neuron-to-Graph (N2G) explanations from activating sequences by taking each sequence’s shortest suffix that still activates the latent at at least half the original strength, testing padding-token wildcards and a leading-pad position check, using up to 16 nonzero activations to build the graph and another 16 as recall true positives. Those N2G scores are measurements on their GPT-2 small sparse-autoencoder latents under a 64-token context, not a Bolukbasi artifact rate and not a rate for Claude or for SAEs in general.
Key Claims
- Observed as author framing, not as a re-run: Gao et al. say recognizable patterns on random activations can create an “illusion” of interpretability, in the sense of Bolukbasi et al. 2021, when explanations are overly broad (good recall, poor precision). Their GPT-2 small remark that many latents “seemed interpretable” on random positive activations is anecdotal. They do not restate Bolukbasi’s BERT checkpoint family or corpus contrast, and this review does not reconstruct them.
- Reported from Bills et al. 2023, not re-estimated here: an automated interpretability score that “disproportionately depends on recall” can score an all-“stop” explanation highly even when the activating pattern is the end of “don’t stop” or “can’t stop.” No score value appears in Gao et al.’s paragraph. Their further claim that this failure gets more severe as autoencoder features get sparser is their interpretation, not a tabulated illusion rate.
- Method they state, then apply: N2G explanations are collections of n-grams with wildcards, built from activating sequences by shortest-suffix search, padding wildcards, and an absolute-position check. The half-strength rule is attached to the suffix step. This is an input-pattern explanation, not a causal circuit.
- Measured on their autoencoders, with a stated proxy limit: for GPT-2 small, N2G is generally easier with more total latents and fewer active latents, except that dense latents (including n = 2048, k = 512) are trivially easy. Versus ReLU at the same n and similar L0, they report TopK N2G recall above 1.5 times and precision above 0.9 times ReLU’s, with better F1. Those ratios are binarized activation-agreement scores, not a transferred artifact rate.
In This Guide
What Gao et al. mean by the cited illusion
The paper under review is Gao, Dupré la Tour, Tillman, Goh, Troll, Radford, Sutskever, Leike, and Wu, “Scaling and evaluating sparse autoencoders,” arXiv:2406.04093v1. Section 4.3 is about whether sparse-autoencoder latents have simple explanations that are both necessary and sufficient for activation. The BERT paper enters only as a citation.
Gao et al. write that, anecdotally, their autoencoders find many features with quickly recognizable patterns that suggest explanations when one views random activations (their Section E.1). They then say this can create an “illusion” of interpretability, citing Bolukbasi, Pearce, Yuan, Coenen, Reif, Viégas, and Wattenberg, “An interpretability illusion for BERT,” arXiv preprint arXiv:2104.07143, 2021. The gloss they attach to that citation is narrow and explicit: explanations are overly broad, and thus have good recall but poor precision. In their usage, recall is coverage of the true activations; precision is how often the explanation’s predicted activations are real. A pattern that is easy to name from a handful of positive examples can still fire, under that name, on many cases the latent does not treat as the feature.
Definition as used in this paper. The interpretability illusion, in Gao et al.’s citation gloss, is not a claim that a recognizable pattern is meaningless. It is the failure mode in which a pattern suggested by activating examples is too coarse: it recalls the activations that prompted it and also many others, so precision is poor. Gao et al. do not, in this version, quote Bolukbasi’s dataset names, name a BERT checkpoint family, or describe the corpus contrast that produced the original artifact.
That omission is a limit of the gloss, not a license to fill it in. The bibliographic version Gao et al. print is the 2021 preprint arXiv:2104.07143. Checkpoint identity and the exact corpus contrast are not supported by the Gao text, so they are not reconstructed here. Nothing in Section 4.3 is a replication of Bolukbasi et al., and no artifact rate from that paper is carried forward.
Nearby BERT systems are different objects from that citation. Qi, Pan, Hong, Yang, van den Hengel, and Wu, arXiv:2104.04167v2, propose an object-and-room informed sequential BERT for indoor vision-and-language navigation that encodes instructions and object regions at the same grain, and includes a relative-direction loss plus room-type losses for the next step and the goal location. Muhammad Bilal Zafar, arXiv:2507.01991v1, fine-tunes bert-base-uncased as FinAI-BERT on a balanced set of 1,586 sentences from 669 U.S. bank annual reports (2015–2023) to classify whether a sentence discloses AI, and uses SHAP token attribution on those sentence-level predictions.
The viewing condition they flag is observational. Section E.1 says the majority of latents in their GPT-2 small autoencoders seemed interpretable even on random positive activations, and that ablation effects were typically predictable from the activation conditions. Those are subjective impressions. The same paper’s limitations section says a large fraction of random activations, especially in GPT-4, are not yet adequately monosemantic. Recognizability on a sample of positives is therefore not, on their own account, evidence that an explanation is precise or that the latent is monosemantic. They also do not treat the illusion sentence as a causal intervention: they do not ablate, patch, or swap corpora to reproduce Bolukbasi’s result inside this PDF.
The Bills et al. score they use as the example
Immediately after the gloss, Gao et al. point to one automated-interpretability failure from Bills et al. 2023, “Language models can explain neurons in language models” (OpenAI Blog). What Gao et al. attribute to that score, and the only example they give, is this: Bills et al. propose an automated interpretability score that disproportionately depends on recall. They find a feature activating at the end of the phrase “don’t stop” or “can’t stop,” but an explanation activating on all instances of “stop” achieves a high interpretability score.
The mechanism of that failure matches the gloss. The true activating condition, as Gao et al. report it, is a short preceding context plus “stop.” An explanation whose condition is the token “stop” still covers those true cases, so recall can be high. It also covers “stop” in other contexts, so precision can be poor. If the published score weights recall much more than precision, the broad explanation is rewarded. Gao et al. do not give the numeric score, the simulation model, the subject model, the layer, or the number of examples behind that sentence. Those details live in Bills et al., which this paper does not reanalyze. In particular, Gao et al.’s paragraph does not say that the example was a GPT-2 XL neuron or a GPT-4 simulation; adding that would go beyond the citation as printed.
They then write that as they scale autoencoders and the features get sparser and more specific, this kind of failure becomes more severe. That sentence is their expectation about SAE latents, placed after the Bills example. It is not a count of false explanations, and it is not a measured rate from BERT, Claude, or any SAE width. The empirical work that follows is a cheaper substitute for the expensive part of the Bills-style evaluation: they say precision is extremely expensive when simulations use GPT-4 as in Bills et al., so they do not estimate precision for arbitrary English explanations. They say they would like to make that approximation more tractable later. That wish is proposed, not done.
How they build Neuron-to-Graph explanations
The substitute is what they call an improved version of Neuron to Graph (N2G), citing Foote, Nanda, Kran, Konstas, Cohen, and Barez, arXiv:2305.19911, 2023. Gao et al. describe N2G as substantially less expressive than an English explanation and much cheaper. Its output is a collection of n-grams with wildcards. The extracted text does not itemize which steps differ from Foote et al.; the construction below is the procedure Gao et al. state for the version they use.
Construction starts from sequences that already activate the latent. It does not start from a human label or from a random corpus search for a concept name. For each chosen activating sequence they find the shortest suffix that still activates the latent, with at least half the original activation strength. That half-strength criterion is the footnote on the suffix step only. Shortening is an input intervention on the sequence, used to propose a token pattern. It tests whether a suffix is sufficient to keep the latent on under their threshold. It does not by itself show that the suffix is the model’s only cause of the activation, or that earlier tokens are irrelevant when the threshold is not met.
They then check whether any position in that n-gram can be replaced by a padding token, which inserts a wildcard. Separately, they check whether the explanation should depend on absolute position by seeing whether inserting a padding token at the beginning matters. The extracted construction paragraph does not repeat the half-strength rule for the wildcard or leading-pad checks, so that threshold should not be assumed to govern those tests. The resulting object is a token-pattern explanation: fixed tokens, wildcard slots, and a possible dependence on absolute position. When they later simulate activations, they refer to a value stored at a node in the trie, which is how this n-gram collection is being used, not a second human description.
Sampling is small and asymmetric. They use a random sample of up to 16 nonzero activations to build the graph, and another 16 as true positives for computing recall. “Up to 16” means the build set can be smaller. They do not, in that paragraph, state the sample size used for precision. Precision and recall here are binarized. Figure 25 defines recall as P(n2g > 0 | act > 0) and precision as P(act > 0 | n2g > 0): among truly active tokens, the fraction the explanation marks active, and among tokens the explanation marks active, the fraction that truly activate. A footnote in Section 4.3 says these metrics care only about binarized values, so a weak and a strong activation count the same once both are positive.
What the build step can and cannot catch. Because the graph is fit from positives and then scored on binarized agreement, a shortest suffix of “stop” with a wildcard in front would be the N2G-shaped version of the Bills failure: high recall on “don’t stop” and “can’t stop,” poor precision on other “stop” tokens. Gao et al. do not say they ran N2G on that Bills feature. The example motivates the metric; it is not an N2G output reported in this paper.
Section 4.4 is a further use of the same explanations, not a change to how the graph is built. They simulate autoencoder latents from N2G, decode, and measure downstream loss, so that both precision and recall matter in proportion to how much the latent contributes. N2G produces a simulated value from the trie node. They scale that value by a ratio estimated on a training set of tokens.
Gao et al., Section 4.4, scale the N2G simulated value by
c = E[s a] / E[s2]
s is the simulated value, a is the true activation, and the expectation is estimated over a training set of tokens. They describe this scale as computed “to minimize variance explained.” They do not state centering, units beyond the activation values, a numeric example, or rounding. No independent evaluation of that ratio is reported in the extracted text.
On that layer-8 GPT-2 use of the scaled simulations, Figure 8’s caption says N2G explanations of autoencoder latents improve downstream loss as n grows and k shrinks, and the body says they can explain more of GPT-2 small than bigrams alone. The figure is not transcribed with numeric losses here, so no loss value is stated. This downstream check is an intervention on the forward pass: explained latents replace true latents before decoding. It is still only as expressive as the n-gram trie. A better downstream number means the binarized-and-scaled patterns account for more of the loss, not that the English concept a person might name from the same activations is correct.
Model, layer, and what the scores show
The N2G numbers in Section 4.3 are for GPT-2 small autoencoders, not for a BERT checkpoint and not for the 16-million-latent GPT-4 autoencoder trained elsewhere in the paper. The default GPT-2 small setup in Section 2.1 is residual-stream activations at layer 8, context length 64 for all experiments, with the mean over the model dimension subtracted and inputs normalized to unit norm before the autoencoder. Appendix B says that, unless otherwise noted, autoencoders were trained on the residual activation directly after layernorm, with layernorm weights folded into the attention weights. Figure 8 explicitly places the N2G downstream-loss comparison at layer 8. A later sweep, Figure 28, uses k = 32 and n = 32768 and finds earlier layers easier to explain with token-pattern N2G, while later layers do better on probe loss and logit-difference sparsity. Section 4.3’s main precision/recall paragraph does not restate the layer in the same sentence as Figure 25; the paper-level GPT-2 small default is layer 8, and the layer sweep is a separate result.
Muchane, Richardson, Park, and Veitch, arXiv:2506.01197v1, use Gao et al.’s TopK sparse autoencoder as a baseline and, on layer-20 residual-stream vectors from Gemma 2-2B, normalize inputs to unit norm but do not subtract a mean vector or include a bias term as Gao et al. do, because they found that choice decreased training stability.
They compare activation rules, not datasets. TopK keeps the k largest latents and trains with reconstruction mean squared error only. ReLU is the L1-penalized baseline. For N2G, they report that at the same number of latents n and similar L0, TopK explanations have significantly better recall, above 1.5 times ReLU, and only slightly worse precision, above 0.9 times ReLU, and therefore better F1 (Figure 24). The figure caption says the same thing in words: higher recall, worse precision, better F1. “Above 0.9 times” means TopK precision is slightly below ReLU’s, not that precision exceeds 0.9. The adjacent sentence says TopK has fewer spurious positive activations than ReLU. That interpretation is theirs. It is not identical to the precision ratio, which moves slightly the other way. Section 2.3’s related claim is that TopK increases monosemanticity of random activating examples by clamping small activations to zero, with Section 4.3 as the pointer. The evidence they actually print for that pointer is this N2G comparison plus the qualitative plots, not a human audit of false explanations.
Figure 25’s caption states the average trend and the exception: the average latent is generally easier to explain as k decreases and n increases, but n = 2048, k = 512 latents are easy to explain because many activate extremely densely (Section E.5). The body says dense token patterns are trivial to explain, and that in general more total latents and fewer active latents are easiest for N2G. They also state that GPT-2 small’s model dimension is 768, and that trends break as L0 gets close to that dimension, where latents become dense. Ease of an n-gram explanation in that regime is a density artifact, not evidence of a sharp feature. Figure 7 illustrates the metric on two latents without making the dense/sparse point: one has precision 0.97 and recall 0.56; the other has precision 0.06 and recall 0.05. The caption’s key is that green marks ground-truth activation and underline marks N2G-predicted activation. The extracted sequence text is too corrupted to support a feature name, so none is assigned here.
| Question | What Gao et al. state | What it is |
|---|---|---|
| Illusion when a pattern suggests an explanation | Overly broad explanations: good recall, poor precision; citation to Bolukbasi et al. 2021, arXiv:2104.07143 | Citation gloss and author interpretation. Checkpoint and corpus contrast are not restated. No artifact rate. |
| Bills et al. score example | End of “don’t stop” or “can’t stop” versus an all-“stop” explanation that still scores high because the score depends disproportionately on recall | Example reported from Bills et al. 2023. No numeric score, model, layer, or re-run in this text. |
| N2G from activating sequences | Shortest suffix at ≥ half strength; padding wildcards; leading-pad position check; up to 16 build sequences; another 16 recall positives | Method they describe and apply to GPT-2 small SAE latents. Binarized agreement with activations, not a causal feature proof. |
Do not export a rate. Gao et al. never print a Bolukbasi artifact rate, and they do not measure one for Claude or for SAE latents. Templeton et al. 2024 appears in this PDF for other SAE facts, such as dead latents, not as a replication of the BERT illusion. TopK-versus-ReLU N2G ratios are about their GPT-2 small autoencoders. They are not a false-explanation rate for every SAE width, every layer, or every model family.
How this evidence was produced
The supporting text is the authors’ own methods and evaluation write-up in arXiv:2406.04093v1, especially Sections 2.1, 4.3, and 4.4, Figure 7, Figures 24 and 25, Figure 8, and the limitations list in Section 6. For the empirical N2G claims, they train sparse autoencoders on language-model residual-stream activations, read latents that fire, compress each activating sequence to a short token pattern, and score binarized agreement between that pattern and held positive or predicted-positive tokens. Suffix shortening and padding replacement are input checks used to form the explanation. The Section 4.4 downstream number is a different intervention: simulated latent values are decoded and substituted into the residual stream at layer 8 of GPT-2. The illusion paragraph and the Bills example are not produced by either procedure. One is a citation plus an anecdotal observation; the other is a report of another paper’s score. This PDF therefore does not establish Bolukbasi’s experimental result, does not validate Bills et al.’s scorer, and does not show that a high N2G F1 is a correct English feature explanation.
Limits and counterevidence
The citation boundary is the main limit. Gao et al. name the Bolukbasi preprint and characterize the failure as broad explanations with good recall and poor precision. They do not identify the BERT checkpoint family, the two corpora, or any percentage of neurons that flipped interpretation. A review that supplied those facts from outside this PDF would no longer be an account of Gao et al.’s gloss. Until that paper is read on its own, those conditions stay unknown here, and no SAE or Claude result should be described as the same artifact rate.
N2G’s own limits are stated by the authors. Section 6 says that although they use N2G for computational efficiency, it captures only very simple patterns, and that more expressive explanation methods still need to be cheap enough to simulate if precision is to be estimated. A high binarized score can mean a dense or local token habit, as with k = 512, rather than a specific feature. The metrics ignore activation magnitude; Section 4.4 is their attempt to put magnitude back in, and it inherits the trie. Context length is 64 tokens for all experiments, and they say that may be too few tokens to exhibit the most interesting GPT-4 behaviors. That caveat is about GPT-4 behavior under this context, not a finding that GPT-2 small N2G scores would reverse at longer context. They do not report that test.
Subjective interpretability cuts both ways. Section E.1 reports that GPT-2 small latents often looked interpretable on random positives, which is the viewing setup the illusion warning is about. The limitations section reports that many random activations, especially in GPT-4, are still not adequately monosemantic. Neither passage is a precision audit of the sort Bills et al.’s score failed to provide. Gao et al. also give no conflicting external replication of the N2G ratios. The TopK precision result is slightly worse, not better, than ReLU at matched n and similar L0, so it should not be summarized as “TopK explanations are more precise.”
No replication of Bolukbasi et al. is proposed here. Running one would require that paper’s checkpoint, datasets, and corpus contrast, which Gao et al. do not supply. The supported conclusion is narrower: recognizable activating patterns are, on Gao et al.’s account, an unsafe basis for a feature explanation; the concrete scoring failure they cite is the recall-heavy “stop” example; and the explanations they actually build are small N2G n-gram graphs fit from activating sequences under the suffix, wildcard, position, and sample rules above.
- What Gao et al. mean by the cited illusion
- The Bills et al. score they use as the example
- How they build Neuron-to-Graph explanations
- Model, layer, and what the scores show
- How this evidence was produced
- Limits and counterevidence
Sources
