Vision Sparse Autoencoder L0

Why Vision SAE L0 Exceeds Language SAE L0

L0 counts active latents per token

Section 5.1 of Prisma, arXiv:2504.19475v3 states the measurement rule in one sentence: “High-quality SAEs balance low sparsity, measure by L0 (number of active SAE latents per token), with high explained variance.” L0 is therefore the sparsity side of a two-term criterion. A larger L0 means more dictionary features are active on that token. It does not, by itself, say what fraction of the dictionary is active, and it does not say that the reconstruction is acceptable.

That expression formalizes the prose definition. The PDF text used here does not display this equation. Here x is one language token, or one image patch when the paper reports vision L0 per patch. The index i runs over dictionary features, and z_i(x) is the activation of feature i. The paper does not state whether “active” means exactly nonzero or above a positive cutoff. Until that threshold is known, L0 values from different codebases are not guaranteed to be the same statistic.

Stated ranges for GPT-2 Small and CLIP-B/32

The comparison the paper actually prints is in Sec. 5.1. It is a comparison of two summaries, not a paired experiment.

The sentence is explicit that the language interval is for “language models like GPT-2 Small,” with the citation pointing at GPT-2 small residual-stream SAEs. Prisma does not identify the GPT-2 checkpoint, layer subset, dictionary size, L1 or top-K setting, dataset, or activation threshold behind 12–74. Those conditions live in the cited post. This article does not re-audit that post’s tables; only Prisma’s citation is in the source text used here. The same limit applies to K = 32–256: Prisma attributes it to Gao et al., arXiv:2406.04093, and does not show that it was measured on the same SAEs as the 12–74 interval.

The vision side is also a summary, not a single SAE. “Approximately 500+ per patch in CLIP-B/32” is tied to reference [35], whose own tables are not in this PDF, and to “SAE evaluation tables in Sec. 10,” which are Prisma’s. The paper says those vision SAEs retain “a comparable explained variance to language.” Sec. 10, as printed, has no language explained-variance column, so that comparability is an author statement rather than a number a reader can subtract from the tables.

The paper calls the vision L0 “significantly higher.” The extracted Sec. 5.1 text does not report a hypothesis test, confidence interval, or standard error on the cross-modal gap. “Significantly” is the authors’ qualitative description of the cited and tabulated counts.

A further vision figure in the same section must stay separate. Analysis of a pretrained SAE on the Kandinsky encoder, Prisma reference [12], “reveals an extreme L0 of 122,631 per CLS token.” That is not CLIP-B/32, not a per-patch figure, and not part of the 12–74 versus 500+ comparison. The paper cites it as an extreme case that was still useful for diffusion steering, which is author-reported effectiveness, not a sparsity optimum.

CLIP-B/32 model, layers, and training controls

The disparity prose says CLIP-B/32. The Sec. 10 tables it points to are labeled CLIP-ViT-B-32. The supplement’s model list calls the released suite CLIP-B-32 and says sparse coders were trained on all twelve residual-stream layers. The paper treats these as the tables behind the CLIP-B/32 sentence. It does not print a Hugging Face or OpenCLIP revision, weight hash, image resolution, or a statement of whether “approximately 50 patches” includes the CLS token. Those identifiers are unknown from the text used here.

Sec. 4 states the training controls for the released CLIP and DINO base SAEs, and for CLIP transcoders, all trained on ImageNet-1k:

• Expansion factor ×64, mapping a 768-dimensional activation to 49,152 features. Those two statements match: 768 × 64 = 49,152.
• Encoder weights initialized as the transpose of the decoder weights.
• One epoch, Adam, batch size 4,096. Learning rates were swept from 1×10⁻⁵ to 1×10⁻¹, with cosine annealing after a 200-step warmup. The evaluation tables do not print the learning rate selected for each row.
• L1 coefficients swept over [10⁻¹¹, 1]. The tables do print the coefficient used for each vanilla row.
• Ghost gradients as an auxiliary loss to limit dead features.
• For CLIP, three token configurations: all patches, CLS-only, and spatial tokens; standard ReLU and top-K (k = 64) variants. The supplement also lists a spatial top-K with K = 128. A K = 128 metrics table is not among the extracted Sec. 10 tables.

Other codebase options, activation normalization, dead-feature resampling, early stopping, are described as available. Sec. 4’s “key training config” list does not say they were enabled for these runs. JumpReLU and Gated SAEs are implemented in the library; Table 1’s released CLIP suite used for these numbers is ReLU and top-K, not those variants.

There is no matched language control. Prisma does not retrain GPT-2 Small under this recipe, does not hold dictionary size fixed, and does not intervene on patch size, token type, or information density to explain the gap. The L0 comparison is observational across cited language SAEs and these vision SAEs, plus author interpretation. The evaluation split for the Sec. 10 L0 numbers is not stated separately from ImageNet-1k training in the extracted text.

What the CLIP-ViT-B-32 tables print

Sec. 10 is where a single “500+” becomes a family of statistics. The spans below are the minimum and maximum of the named column as printed, not ranges the paper computes. Sublayer is mlp out or resid post as indicated by the extreme row. Layers run from 0 through 11.

For the all-patch vanilla column that sits closest to a “per patch” average, every printed Avg L0 is above 500. The lowest, 604.44, is at layer 0 mlp out with l1 coefficient 1×10⁻⁵ and 98.7% explained variance. The highest, 2068.3, is at layer 4 resid post with the same l1 coefficient and 98.0% explained variance. Across the Table 4 rows whose printed l1 coefficient is 1×10⁻⁵, explained variance runs from 97.6% to 98.7%. High raw L0 in that subset is not an artifact of a collapsed reconstruction. Two rows use 5×10⁻⁵ instead: layer 2 resid post (Avg L0 717.84, 90.6% explained variance) and layer 11 mlp out (Avg L0 748.14, 89.7% explained variance).

That still does not make “approximately 500+ per patch” a complete description of Sec. 10. Avg CLS L0 inside the same all-patch SAEs includes 10.11, 27.39, 36.92, and 40.46, all at or below the cited language band of 12–74, and also 1337.0. The 10.11 row is the higher-penalty layer 2 resid-post SAE, so it is not matched on L1 to the 1×10⁻⁵ rows. CLS-only vanilla SAEs (Table 5) stay above that language band on Avg CLS L0, from 314.13 to 934.83, but their explained variance is printed as 65% to 96%, including 65% at layer 11. Spatial vanilla Avg L0 is mostly about 758 to 1008; the minimum 88.80 is layer 10 resid post at l1 coefficient 1×10⁻⁴ and 72% explained variance, so it is not a like-for-like sparse solution next to the ~99% rows. Token type, layer, sublayer, L1 coefficient, and whether K is fixed all move L0 by more than the gap between 12 and 74.

Relative to the stated dictionary, these raw counts are still a small fraction. Assuming the Sec. 4 width of 49,152 features applies to the Table 4 SAEs, the tables do not reprint width per row, 604.44 / 49,152 = 0.012297, or 1.23% when the percentage is rounded to two decimal places, and 2068.3 / 49,152 = 0.042080, or 4.21% at the same rounding. Prisma compares raw L0 counts, not these fractions. The fraction cannot be compared with GPT-2 Small here, because Prisma does not print that SAE’s dictionary size. A higher per-token L0 does not mean the vision code is dense in its own dictionary.

Table 6 is the other regime. With k = 64, Avg CLS L0 is 64 by construction, which sits inside the cited language top-K band of 32–256. Printed explained variance on those rows runs from 84% (layer 11) to 96% (layers 1 and 2). The authors still say their top-K SAEs at K = 64 and K = 128 “show polysemantic features.” The extracted text does not define the polysemanticity assay (manual inspection, an automated score, or something else). That judgment is author interpretation. It is not a second measured L0.

Four explanations the paper labels as proposals

After stating the disparity, Sec. 5.1 says: “We propose four potential explanations for this disparity.” The verb is “propose.” None of the four is presented as a result of an intervention that isolates it. The section ends by calling the discrepancy an open question. The four labels below are the paper’s.

Information-density differences

The paper’s proposal is that visual inputs “may have an inherently denser distribution than language, lack the power-law distribution of natural language, and may require more extensive preprocessing.” The density and preprocessing clauses are hedged with “may.” The power-law clause is not: the paper prints “lack,” not “may lack.” No information-density measurement, power-law fit, or preprocessing ablation is reported in the extracted text. This proposal can be stated. It cannot be scored from Prisma’s tables.

Patch granularity and the factor-of-20 rescaling

The paper states that CLIP-B/32 processes approximately 50 patches, compared with GPT-2’s 1024 tokens, “a 20× difference.” It then says: “When normalized (25 per patch), vision L0 becomes more comparable to language.” It adds that the relationship between patch size and L0 “warrants further investigation.” The 25 is a proposed rescaling, not a second measured CLIP L0, and not a change in the definition of L0.

The arithmetic that reproduces the printed 25 uses the rounded factor, not the unrounded token ratio. Inputs are the paper’s own figures: a vision L0 of 500 (the bottom of “approximately 500+,” not a table cell), approximately 50 patches, and 1024 tokens.

• 1024 / 50 = 20.48, which the paper rounds to a 20× difference.
• 500 / 20 = 25 exactly. That matches the printed “25 per patch.”
• Using the unrounded ratio instead: 500 × 50 / 1024 = 25,000 / 1,024 = 24.4140625, which is 24.41 to two decimal places and 24 to the integer precision of “25.” The printed 25 matches division by the rounded 20× factor only.

Assumptions that the paper does not discharge: that 500 is the right vision L0 to insert; that dividing an already per-patch count by (language tokens / vision patches) is the right normalizer; and that the result is still “per patch” in the same sense as the measured L0. L0 is already per position. This division changes the comparison convention. It does not measure a different patch size. The paper does not derive the normalizer, and the last sentence of the proposal asks for further investigation.

The rescaling is sensitive to which vision L0 is inserted. Applying the paper’s own 20× rule, not a rule Prisma applies to these rows, to the Table 4 Avg L0 extremes: 604.44 / 20 = 30.222, which rounds to 30 at the nearest integer and falls inside 12–74; 2068.3 / 20 = 103.415, which rounds to 103 and falls above 74. Under that proposed normalizer, “more comparable to language” is true for the prose figure of 500 and for the lowest all-patch Avg L0, and false for the highest. That sensitivity is a calculation on printed numbers, not a Prisma result. It is why 25 should not be cited as the vision L0 range.

Patch-type specialization

The paper proposes that spatial patches and CLS tokens have distinct distributions, citing references [35], [46], and [65], and that the CLS token’s high L0 “encapsulates a global image representation.” It then states two normalized figures without showing the formula: “When normalized (L0 ≈ 479 per patch) and adjusted to language-scale tokenization (L0 ≈ 120), this aligns with language SAE sparsity.” Both 479 and 120 are inside the proposal. The extracted text does not identify the SAE, layer, or algebra that produces them. They are not cells in Tables 4–7 (the nearest Table 4 Avg CLS L0 to 479 is 494.3 at layer 7 resid post, which is not 479). Dividing 479 by 120 is about 4, not the 20× factor from proposal 2, so this is a different adjustment. Because the formula is absent, this article does not reconstruct it.

What the paper does measure, and then interprets, is dead versus alive features. The Fig. 2 caption says: “The percentage of alive features shows that CLS tokens require more features in deeper layers while spatial tokens show declining utilization, showing how information shifts from spatial patches to a global representation. Results are from the vanilla CLS SAEs (Tab. 5).” Table 5’s printed % alive features for those CLS-only vanilla SAEs is 4.33, 2.82, 2.54, 4.49, 15.75, 16.80, 29.46, 51.68, 82.07, 93.68, 99.98, and 99.996 from layers 0 through 11. The early layers are not a monotone rise: the minimum is 2.54% at layer 2, and layers 0–3 stay between 2.54% and 4.49%, after which the rate climbs to 99.996% at layer 11. The caption’s spatial decline cannot be read off Table 5, which prints CLS metrics only. The figure image was not available in the text used here, so the spatial series remains a caption claim.

Alive rate is not L0. In the same Table 5, Avg CLS L0 is 934.83 at layer 0 and 386.09 at layer 11; it does not track the rise in % alive. Ghost gradients were part of the stated training recipe, and large layerwise differences in alive rate remain. The authors write that this pattern “suggests” optimal top-K varies by token type and depth, with CLS tokens requiring higher K in later layers. “Suggests” is the right strength. They do not report a K sweep that estimates that optimum.

Domain-specific optimization

The fourth proposal is that current vision SAE training is adapted from language interpretability and that this may limit achievable sparsity, while vision-specific methods “are yet to be developed.” The descriptive half fits Sec. 4: tied encoder–decoder initialization, L1, ghost gradients, and top-K are language dictionary-learning tools the paper cites and uses. The causal half, that this transfer is what keeps vision L0 high, is not tested. There is no ablation of a vision-specific trainer against this recipe at matched explained variance. “Yet to be developed” is the authors’ statement about the field, not a search result.

Optimal vision SAE sparsity is left open

The paper says this twice, at two strengths. Opening Sec. 5, it says the CLIP ViT representations “appear much less sparse than in language, leaving optimal sparsity for vision SAEs an open question.” Closing Sec. 5.1, it says: “This apparent sparsity discrepancy across modalities remains an open question, and we hope these insights serve as a starting point for further research.” Sec. 5 is titled as preliminary analysis. The conclusion repeats that vision representations are less sparse than language models and invites further work. Neither passage selects an L0 target, a K, or one of the four proposals.

The open item is specifically the operating point: which L0, at which explained variance, for which token type and layer, counts as the right vision SAE. Table 6 shows that setting K = 64, inside the language band cited from Gao et al., is feasible as a constraint and still leaves the authors calling features polysemantic. That does not identify a better K. It also does not say that vanilla L0 must stay above 500. The tables show L0 moving with the L1 coefficient and the token configuration, including CLS statistics inside the cited language interval.

Limits, table counterevidence, and a proposed matched comparison

The main limit is internal to the tables the paper cites. “Approximately 500+ per patch” is a fair floor-style reading of Table 4’s all-patch Avg L0, all of which exceed 500 at high explained variance for the 1×10⁻⁵ rows. It is not a fair reading of every L0 column in Sec. 10. Avg CLS L0 on all-patch SAEs can sit inside 12–74. CLS-only vanilla L0 stays in the hundreds, but explained variance falls as low as the printed 65%. Spatial L0 near 90 occurs on a row with 72% explained variance and a much larger L1 coefficient. Top-K CLS L0 is 64 because K is 64. Any claim that vision SAE L0 exceeds language SAE L0 has to name the column, the token, the penalty or K, and the reconstruction level. Prisma’s prose comparison does not do all of that.

Other bounds: dictionary width is unmatched to the cited GPT-2 SAEs, so raw L0 is not a sparsity fraction comparison. The active-latent threshold is unstated. The CLIP checkpoint hash and image size are unstated. No significance test is reported. Reference [35] and the Bloom tables were not independently replicated in the text used here. Table 9 is a DINO-B suite and is not a substitute for the CLIP-B/32 range. The Kandinsky L0 of 122,631 per CLS token is a different SAE. Sec. 5.2’s finding that some SAE reconstructions reduce cross-entropy is a different observation; the paper does not use it as evidence for any of the four L0 proposals.

Proposed, not run in Prisma and not run here: a matched comparison would use one SAE codebase; state the active-latent threshold; report L0 per position and L0 divided by dictionary size; stratify by layer and by CLS versus spatial patch; and lock a reconstruction target, such as explained variance, before comparing L0. Model name, checkpoint, image size, and patch count would be part of the report. The factor-of-20 rescaling would be a hypothesis under test, not a correction, and would be compared with at least one alternative normalizer, such as the dictionary fraction above. Until that exists, the supported answer is the operational definition, the cited 12–74 per token versus the stated approximately 500+ per patch, the wider and conditional Sec. 10 counts, four explicitly labeled proposals, and an optimum the authors leave open.

1. L0 counts active latents per token
2. Stated ranges for GPT-2 Small and CLIP-B/32
3. CLIP-B/32 model, layers, and training controls
4. What the CLIP-ViT-B-32 tables print
5. Four explanations the paper labels as proposals
6. Optimal vision SAE sparsity is left open
7. How this evidence was produced
8. Limits, table counterevidence, and a proposed matched comparison