In arXiv:2606.03990v1 (Dravid, Bahri, Efros, and Gandelsman; preprint, 2 June 2026), Rosetta Neurons are the class adopted from Dravid et al. (2023): neurons whose activation patterns are similar, and recur, across independently trained models. The working units in this paper are Transformer MLP coordinates retained as mutual nearest-neighbor pairs under Pearson correlation of aligned token activations, default top-1.
Treated as a scaling observable, that recurring population is assigned three properties, universality, selectivity, and specialization, and is identified in language and vision models spanning 80 million to 30 billion parameters. The scaling result is about count versus share: across family comparisons, the number of discovered correspondences follows a sublinear power law in average neuron count, with fitted exponents about 0.5–0.7 and reported R² around 0.99, so the absolute number grows while the fraction of all neurons shrinks. That statement is an ordinary-least-squares description of matched counts under the paper’s data and matching conditions. It is not an output of the later capacity model.
The phenomenological capacity model is placed after that fit, in Section 4.3, as an account of why the count is sublinear. It does not estimate the reported exponents. This is what the v1 preprint reports, not an independent replication. The same preprint reports that DINOv3 does not show a clear version of the law, and it does not tabulate a separate fraction-versus-scale series.
Key Claims
- Observed definition. The adopted notion is cross-model recurrence of activation patterns. The counted units are mutual nearest-neighbor MLP pairs (or, on collapsed curves, neurons that survive intersection across those matches), not attention heads and not sparse-autoencoder features.
- Observed assignment. The paper assigns the recurring population three scale properties, universality, selectivity, and specialization, over models it summarizes as spanning 80M to 30B parameters. The Section 4 law quantifies population size; it does not fit separate power laws for selectivity or specialization.
- Observed count-versus-fraction result. |R| = c xα in average neuron count x, with α sublinear (prose range approximately 0.5–0.7; Figure 3 prints exponents from 0.49 to 0.69). Absolute |R| rises; because α < 1, the share |R|/x falls. Fits use log-log ordinary least squares. Reported R² values are “around 0.99,” not a per-curve table.
- Author account, placed after the fit. Section 4.3’s capacity-allocation model is offered to explain that sublinear law and to predict later polarization. Its scaling form is Θ(N1/β) under β > 1. The paper does not report those empirical α values as estimates of 1/β.
- Not established. A single universal exponent, a causal intervention on scale, derivation from gradient dynamics, or coverage of every training recipe. Untrained and input-permutation controls lack the law; DINOv3 is an author-reported exception.
In This Guide
Units adopted from Dravid et al.
Dravid, Bahri, Efros, and Gandelsman separate a previously named class from the filter they use to count it. The abstract calls Rosetta Neurons “a previously characterized class of neurons whose activation patterns are similar across independently trained models (Dravid et al., 2023).” On page 2 they “adopt the notion of Rosetta Neurons from Dravid et al. (2023): neurons whose activation patterns recur across independently trained models.” The 2023 citation, as listed in this preprint, is Dravid, Gandelsman, Efros, and Shocher, “Rosetta neurons: Mining the common units in a model zoo,” ICCV 2023, pp. 1934–1943. Related work says that paper identified shared Rosetta Neurons across diverse vision models. This preprint does not restate the 2023 matching equations, so the operational filter below is this paper’s, not a reconstructed 2023 protocol.
Section 3 restricts the units to MLP neurons in Transformers. For hidden state h at token t in layer ℓ, the MLP intermediate activation is an elementwise nonlinearity applied after an affine map. Each coordinate of that intermediate vector is one neuron, indexed by layer and channel. Attention heads are outside the count. The paper does not convert these coordinates into sparse-autoencoder features, and the neuron totals below should not be read as feature counts from a different model or dictionary width.
Working unit. A neuron is one coordinate m(ℓ,c)t of m(ℓ)t = ϕ(W(ℓ)in h(ℓ)t + b(ℓ)in).
Pairwise similarity is Pearson correlation of those activations on aligned tokens. For neurons u and v, sim(u, v) = (1/N) Σi,t [(mut(xi) − μu)/σu] · [(mvt(xi) − μv)/σv], where N is the number of aligned tokens, and μ and σ2 are empirical mean and variance. A pair enters R(A, B) only if each neuron is among the other’s top-k neighbors. Default k = 1. The paper then “interpret[s] the pairs of neurons in R(A, B) as common units across models, and refer[s] to them as Rosetta Neurons.”
Alignment is part of the definition’s conditions, because the correlation is undefined until positions match. For language, shared UTF-8 byte boundaries define tokenizer-independent spans, and activations of tokens overlapping a span are mean-pooled. For vision, class or register tokens are removed and patch grids are bilinearly resampled to a shared spatial grid, using the maximum of the two map sizes. Figure 2 illustrates the resulting mutual nearest neighbors on the same high-activating inputs.
What is being counted. For a two-family comparison, |R| is a correspondence count under mutual nearest neighbors, not the number of neurons in one model. When curves share a model, the paper intersects matches inside that model. That intersection is a stricter count, neurons shared across more than two families, not the same object as a single pairwise match list. Both are fit with the same power-law form.
Three properties and the 80M–30B span
Page 2 assigns the recurring population three neuron-level properties across scale: “universality, selectivity, and specialization.” Figure 1 states the questions those names carry. Universality asks how the recurring population scales. Selectivity asks whether recurring neurons become increasingly monosemantic relative to polysemantic neurons. Specialization asks which features of the data distribution Rosetta Neurons encode at different scales. The same page then says the authors “treat Rosetta Neurons as a scaling observable, identifying recurring cross-model units in language and vision models spanning 80M to 30B parameters.”
Those three properties are not three fitted exponents. Universality is the property operationalized by the Section 4 count. Selectivity and specialization are later measurements on the same identified population, using vocabulary-space excess kurtosis, a vision-language-model judge, document-type firing, and one code-filtering case study. They are not inputs to |R|.
The 80M–30B figure is the paper’s summary parameter span, not the regression variable. The abstract splits it into language models up to 30B parameters and vision models up to 5B parameters. Section 4.2 states the fit ranges more narrowly: language laws span roughly 100 million to 30 billion parameters, about 40 thousand to 2 million neurons; vision laws span roughly 80 million to 5 billion parameters, or 40 thousand to 600 thousand neurons. The listed language pairs include OPT-30B with Qwen2.5-32B, so the abstract’s “30B” is the paper’s rounded upper summary, not a claim that every listed checkpoint is at most 30 billion parameters. Exact parameter totals for each checkpoint are not recalculated here.
Language families are Pythia, GPT-2, OPT, and Qwen2.5, restricted to pretrained checkpoints that have not undergone post-training. Vision discriminators are OpenCLIP, DINOv2, and Pixio; the generative side is one-step diffusion on Diffusion Transformers from the Pixel Mean Flow, Sana, and Flux families. Points on a curve pair one model from each family at approximately matched scale. The paper says the vision trend appears under contrastive image-text supervision, self-distillation, masked autoencoding, and diffusion or flow-based generative modeling. That is a description of the studied families, not a test of every objective.
Sublinear count law and shrinking fraction
The observable’s scaling result is stated before any capacity model. Section 4.1 sets |R| = c xα, where |R| is the number of discovered correspondences and x is the average total neuron count of the approximately scale-matched models. The exponent α is the quantity of interest; c depends more strongly on family and setting. The curve is fit by ordinary least squares in log-log space. This is a cross-model association between scale and a matching count. It is not an intervention that changes width inside one training run.
Empirical law, Section 4.1–4.2. |R| = c xα, estimated in log |R| versus log x.
Share implied by that law, not a second fit. f(x) = |R| / x = c xα−1. If α < 1, then α−1 < 0, so f decreases as x increases. No endpoint fraction is computed here: the preprint states the shrinking share in prose and does not, in the extracted Section 4 text, tabulate |R|/x at each scale. Substituting a tick label from Figure 3 into a printed fit would invent a measured fraction.
The abstract reports the joint result: in separate language and vision analyses, “the population of Rosetta Neurons follows a sublinear power law in model size, growing in absolute number but occupying a shrinking fraction of the total neuron count.” Page 2 repeats the same contrast in neuron count: larger models contain more shared units, yet those units occupy a shrinking fraction of the total neuron population. Section 4.2 adds the fit summary. Language and vision curves are “well fit by power laws, with R² values around 0.99,” and “the fitted exponents lie in a narrow and consistently sublinear range, between approximately 0.5−0.7.” The same qualitative sublinearity is reported for collapsed multi-family curves. The paper does not publish one shared α for all families.
Figure 3 prints the individual fits. Family-to-equation pairing below follows the order of equations and legends in the PDF text extraction of that figure; a reader checking the rendered figure should confirm the pairing. The prose range does not depend on that pairing. The lowest printed exponent is 0.49, which sits just under the prose floor of “approximately 0.5.” The highest in this main, default-k figure is 0.69.
| Comparison as ordered in the Figure 3 extraction | Printed fit |R| = c xα | α |
|---|---|---|
| Language: OPT–Qwen2.5 | 12.43 x0.57 | 0.57 |
| Language: Pythia–OPT | 2.20 x0.68 | 0.68 |
| Language: Pythia–GPT-2 | 7.98 x0.54 | 0.54 |
| Language: OPT–Pythia–GPT-2, intersected | 2.43 x0.59 | 0.59 |
| Language: Pythia–OPT–Qwen2.5, intersected | 1.99 x0.60 | 0.60 |
| Vision: Diffusion–DINOv2 | 5.25 x0.55 | 0.55 |
| Vision: Diffusion–Pixio | 8.16 x0.49 | 0.49 |
| Vision: Diffusion–OpenCLIP | 1.13 x0.63 | 0.63 |
| Vision: Pixio–Diffusion–OpenCLIP, intersected | 0.23 x0.69 | 0.69 |
| Vision: DINOv2–Pixio–Diffusion–OpenCLIP, intersected | 0.17 x0.68 | 0.68 |
Matching conditions for these counts are specific. Each language pair uses about 10 million tokens, sequences sampled i.i.d. from The Pile. Each vision comparison follows the 2023 generative-versus-discriminative setup: 50,000 class-balanced images sampled from a diffusion model conditioned on ImageNet-1k labels, then passed through the discriminator. Default k is 1. x is mean MLP neuron count, not parameter count. Axis ticks visible in the Figure 3 extraction, on the order of 105 to 106 neurons for language and 105 to 5×105 for vision, are ticks, not a substitute for the prose spans above.
Two controls argue that the count trend is not produced by the matcher alone. On untrained networks using architecture-specific initialization, three seeds give a marginal number of matches and no systematic increase with size (Figure 4). After permuting one model’s flattened activation positions, which keeps each neuron’s marginal distribution but breaks input alignment, discovered matches fall to roughly 20–100 and lose the power-law trend (Section C.3, three permutations, means with 95% confidence intervals). Those are negative controls on the procedure, not causal estimates of how training creates the matches.
Where the capacity model sits relative to that result
The preprint places the capacity model after the empirical law and uses it to interpret the law, not to compute it. The introduction says the authors “then develop a phenomenological model linking this sublinear scaling to limited neuron capacity (Section 4.3).” Section 4.3 opens from Figure 3: the empirical law raises the question of why Rosetta Neurons grow sublinearly and occupy a shrinking fraction. The section heading calls the same construction an analytical model. Discussion later calls it “a simple account” and says deriving the phenomena from gradient-based training dynamics “remains an important next step.”
Do not substitute the model for the result. Reported α values are ordinary-least-squares fits to matched counts. They are not solved from an estimated feature-importance exponent β. A theoretical 1/β can be sublinear for the same qualitative reason, an exponent below 1, without being the number printed in Figure 3.
The model adopts a capacity-allocation view of superposition: many features compete for limited neuron coordinates; higher-importance features are represented more cleanly; neurons dominated by one shared feature are treated as easier to match across independently trained networks. Assumptions stated for the derivation include a superposition regime in which the number of represented features A(N) is Ω(N) and greater than the neuron count N; a power-law importance spectrum wr ∝ r−β with β > 1; an isolation score sr for the neuron most aligned with feature r; and a Rosetta-detectability cutoff sr ≥ τ with τ > 1, so each detectable feature has a distinct dedicated coordinate in the idealized signal-to-interference model. The allocation objective maximizes Σr wr log(1 + sr) subject to Σr sr ≤ κN. In the continuum solution, the detectability frontier scales as Θ(N1/β).
Model consequence, Section 4.3, not the empirical estimator. Rτ(N) = Θ(N1/β), with β > 1, hence 1/β < 1.
N is neuron-coordinate count in the idealized network. Rτ(N) counts features whose isolation meets τ. β is the assumed importance-spectrum exponent, not a coefficient fit to Figure 3. κ is the constant in the linear isolation budget. The paper’s derivation of the log utility and linear budget is in Appendix D; those steps are part of the account, not additional empirical exponents.
Because this exponent is below 1 whenever β > 1, the model is offered as an explanation of sublinear count growth and of a shrinking share of coordinates. The same allocation profile predicts a Neuron Polarization Effect: mean isolation inside the Rosetta set grows as Θ(N(β−1)/β), while clean isolation in the non-Rosetta tail vanishes under A(N) = Ω(N). Section 5 then reports proxy measurements consistent with that prediction. Those proxies, excess kurtosis of vocabulary-space cosine similarities, and a VLM judge of top-activating images, are separate from the |R| regression. Appendix D.7 is a synthetic packing simulation in which the same matching rule recovers slopes near 1/β under the model’s isolation profile. It checks the model against its own generator. It is not the language or vision scaling result.
What the count evidence is. The Section 4 result is a population correlation: scale-matched, independently trained checkpoints are run on a shared input cache; MLP activations are aligned; mutual top-1 Pearson neighbors are counted; that count is regressed on average neuron count in log-log space. Language caches are about 10 million Pile tokens. Vision caches are 50,000 class-balanced diffusion samples conditioned on ImageNet-1k labels. Language checkpoints are pretrained Pythia, GPT-2, OPT, and Qwen2.5 models, not post-trained models. Vision pairs cross OpenCLIP, DINOv2, Pixio, and one-step Diffusion Transformer generators. No single layer is the probe. Appendix E.3, using Pythia–OPT and Diffusion–OpenCLIP matches, finds Rosetta units in multiple normalized-depth bins and treats that spread as a non-degeneracy check, not as a depth scaling law.
What it is not. It is not a causal intervention on scale, not a within-run ablation, and not a map of all shared computation. The authors interpret recurrence as more than a matching artifact because untrained and alignment-broken controls do not reproduce the law. That interpretation is stronger than the raw correlation, and weaker than an identification of the capacity mechanism. The capacity model is a subsequent phenomenological account. Selectivity, specialization, and the JavaScript filtering case study test other properties of already identified units; they do not generate the exponents in Figure 3.
Limits and counterevidence
The main-text range “approximately 0.5−0.7” is tied to the default k = 1 fits. Section C.2 repeats Pythia–OPT and Diffusion–OpenCLIP matching at k = 2, 5, and 10. Printed exponents stay below 1, but they are not frozen: the extraction lists Pythia–OPT exponents 0.68, 0.75, 0.78, and 0.71, and Diffusion–OpenCLIP exponents 0.63, 0.61, 0.64, and 0.56. The paper’s claim is a narrow sublinear range under a more permissive matcher, not invariance of the 0.5–0.7 interval. Changing k changes which features are counted and changes the prefactor; the authors say it should not change the underlying exponent. The printed ablation shows small movement, including one exponent above 0.7.
DINOv3 is the preprint’s explicit failure case. Repeating discriminative-to-diffusion matching, Section I.1 reports no clear Rosetta scaling law and no monotonic count trend, unlike DINOv2. The authors relate the deviation to DINOv3’s extra constraints on intermediate representations and treat it as evidence that the law depends on data, architecture, optimization, and objective together. That is author-reported counterevidence inside one paper, not an external replication. No independent refit of Figure 3 is cited.
Discussion limits the object itself. Rosetta units are one neuron-level subset. Circuits, attention heads, and subspaces can carry shared computation without appearing as neuron correspondences, so the population is “a tractable observable,” not “a map of all shared computation.” Monosemanticity results use modality-specific proxies and are described as relative population trends. The analytical model is not derived from the training dynamics that produced the checkpoints.
Unknowns that this v1 text does not close: a tabulated fraction at each scale; per-curve R² beyond “around 0.99”; whether later revisions change the printed fits; and whether the 2023 matcher is identical to this Pearson mutual-neighbor filter. Pointwise fractions and any identification of empirical α with 1/β would be new calculations, not results reported in Section 4.2. This account follows the 2 June 2026 preprint and does not add a replication.
- Units adopted from Dravid et al.
- Three properties and the 80M–30B span
- Sublinear count law and shrinking fraction
- Where the capacity model sits relative to that result
- Limits and counterevidence
Sources
