GPT-4 Wrote Explanations for 307,200 GPT-2 XL Neurons and Graded Them by Simulation

Fact-checked by the J-Space editorial team

Updated September 19, 2026

Transformer language models predict the next token by routing activity through stacked MLP blocks, yet for most units there is no reliable account of what the unit computes. OpenAI’s May 2023 research note, released on May 9, 2023, asks whether a stronger model can automatically write explanations of those units and whether the explanations can be scored rather than merely read. The object of study is GPT-2 XL MLP neurons; the explainer and simulator is GPT-4. The model does not explain itself.

This article reconstructs the paper as a method: generate explanations, simulate activations, and score the match to real neuron activations. It clarifies the GPT-4 versus GPT-2 XL division of labor, the 307,200 neuron corpus, public artifacts, and the later shift from raw neurons to sparse features. It does not claim that black-box systems have been explained, that fluent captions are circuits, or that large language models are conscious.

What OpenAI’s Neuron-Explainer Work Actually Is

OpenAI’s 2023 note is a scalable method for labeling MLP units in a smaller subject model, not a dashboard that makes language models transparent and not a demonstration that GPT-4 explained its own neurons. The authors – Bills, Cammarata, Mossing, Tillman, Gao, Goh, Sutskever, Leike, Wu, and Saunders – published the writeup on May 9, 2023, with an interactive HTML paper rather than a PDF-only conference drop. The subject is GPT-2 XL MLP neuron activations after the GELU nonlinearity; the tool is GPT-4 in two roles, explainer and simulator. Large language models here are used as hypothesis generators over another neural network’s internals. That division of labor is the fact most viral summaries invert. GPT-4 never receives its own weights, never inspects GPT-4 MLP units, and never “interprets itself.” It is shown contexts that light up a GPT-2 XL neuron and asked to describe the pattern, then asked to predict activations on held-out text from that description alone. The project sits in the circuits / Zoom In lineage: instrument a unit, collect the conditions under which it fires, and treat the resulting label as a hypothesis about one coordinate of model computations, not as a recovered algorithm. Neuron explanations are not full circuit maps, and they do not resolve debates about machine consciousness or global-workspace dynamics. They are the largest-scale public attempt, at the time of release, to attach human-readable descriptions to internals of a neural network and to score those descriptions against held-out neuron activations rather than against reader plausibility.

Why the work still matters for interpretability engineers is the combination of named authorship, a full-width scan of GPT-2 XL MLP units, and released tooling. OpenAI shipped a research page, an interactive neuron browser covering the GPT-2 XL MLP set, and code in openai/automated-interpretability. That is a reproducible baseline, not a teaser blog. At Jspace.com we treat it as a data point for mechanistic interpretability and for J-Space / Jacobian Lens analysis: a neuron label is a local slice through a high-dimensional causal graph, convenient because every neural network already indexes those coordinates, not because neurons are the uniquely natural basis. Functional subnetworks can resemble distinct circuits, and the “right” features may cut across many units. Still, neurons are where this project starts. The paper explores hundreds of thousands of units across GPT-2 Small, GPT-2 XL, and GPT-3 variants, uses GPT-4 to generate explanations, and evaluates those explanations with correlation metrics. Many interesting neurons appear in the long tail; the vast majority remain mixed, low-scoring, or uninterpretable under a one-sentence caption. The sections below keep that proportion in view. The method is closer to automated microscopy than to an XAI product: it can automatically write explanations at industrial volume, then discard or down-rank captions that fail an activation-match test. It is not a substitute for evaluations, red-teaming, or governance, and the NIST AI Risk Management Framework is the policy frame that treats explainability and interpretability as characteristics of trustworthy AI without treating any single lab pipeline as a safety certificate.

• Object of study: post-GELU MLP neuron activations in GPT-2 family models, primarily GPT-2 XL, with GPT-2 Small and GPT-3 variants used to probe scaling of subject-model difficulty.
• Instrument: GPT-4 as explainer and as simulator, with smaller GPT-family models ablated in those roles to test whether explanation quality scales with explainer/simulator capability.
• Non-claims: GPT-4 does not interpret GPT-4; a fluent sentence is not a circuit; scoring explanations is not mechanistic causal tracing; neuron-level labels are not proof that language models are interpretable end to end.
• Lineage: feature visualization and circuits work (Olah and collaborators), superposition analyses, and later dictionary-learning / sparse autoencoder autointerp that reused generate-and-score on features rather than on raw neurons.
• Release form: research page plus HTML paper plus neuron browser plus GitHub, dated May 9, 2023, rather than an arXiv-first conference paper as the primary artifact.

How GPT-4 Generate Explanations, Simulate Activations, and Score Them

The contribution is not that GPT-4 writes captions for neurons. The contribution is a closed loop in which GPT-4 must generate explanations from real neuron activations, freeze the text, produce simulated activations on unseen tokens, and accept a score only if those simulated activations match the subject neuron closely enough to be useful as a hypothesis. Almost no secondary writeup walks through that third step, which is why readers walk away thinking the result is copywriting. Step 1 selects text excerpts where a neuron fires most strongly, annotates tokens with activation strength, and asks GPT-4 what pattern causes the unit to fire. Step 2 gives GPT-4 only the resulting sentence plus new text and asks it to predict per-token activations. Step 3 compares those predictions with held-out real neuron activations, typically via Pearson correlation after a linear calibration that absorbs scale mismatch. Care is taken to limit overlap between the explanation corpus and the simulation corpus, reducing memorization of the same spans. Automated systems can use capable language models to explain neural activations at a scale no smart human annotator could match neuron-by-neuron, but the loop is only as good as the score. If simulated activations match poorly, the caption is a failed hypothesis even when it reads like a clean feature name. That is the difference between this paper and generic saliency dashboards: scoring explanations is first-class experimental apparatus, not a UI flourish around a few cherry-picked units.

1. Generate explanations. For each neuron, take top-activating contexts, show them to GPT-4 with per-token scores, and ask GPT-4 to describe the neuron’s top activating behavior in natural language.
2. Simulate activations. Freeze that description. On held-out text, GPT-4 predicts a 0-10 activation for each token using only the description, producing a vector of simulated activations.
3. Score the explanation. Correlate simulated activations with real neuron activations. Higher explanation scores mean the sentence constrained GPT-4’s predictions toward the true firing pattern.

The explanation-generation step is where raw numbers become hypotheses about a neuron’s top activating behavior. For a given unit the system selects approximately five 64-token spans from documents where the neuron fires most strongly. Activation magnitudes are normalized to a 0-10 integer scale, with negative values clamped to zero, so GPT-4 can reason about relative strength without floating-point clutter. When a neuron is sparse, activating on a small fraction of tokens, non-zero examples are repeated or emphasized so GPT-4 is not drowned in zeros. The prompt is direct: annotated tokens in, concise pattern description out. An explanation might read that the unit fires on legal-proceeding tokens such as plaintiff, defendant, and ruling. GPT-4 produces code-syntax captions for programming units, date and temporal captions for calendar units, and pop-culture captions when the training distribution supports them, including Marvel comics and Marvel Cinematic Universe vocabulary. Diversity of explanations tracks diversity of pretraining text, not a guarantee that the caption is complete. Sparse highlighting, integer binning, and short outputs are design choices that make GPT-4 usable as a describer; they also compress away token-level quirks that later baselines recover. The same GPT-4 stack is then reused to generate explanations under revision: false positives and false negatives from scoring are fed back so GPT-4 can tighten the sentence. That revision loop is just a new flavor of automated science – hypothesize, test, edit – and it is how higher scoring explanations are obtained without pretending the first caption was mechanistic ground truth.

• Excerpt budget: a handful of top-activating 64-token windows per neuron, drawn from a large corpus, with a separate evaluation mix of top-activating and random passages.
• Normalization: real neuron activations mapped onto a 0-10 integer scale for both the explainer prompt and the simulator’s output space.
• Sparsity handling: extra weight on rare non-zero fires so GPT-4 attends to the neuron’s top activating behavior rather than to the sea of near-zero tokens.
• Prompt style: ask GPT-4 to explain neurons with a short natural-language pattern, not a multi-paragraph essay and not a weight dump.
• Revision: return the worst mismatches so GPT-4 can generate explanations that mention counterexamples (simile versus technical comparison, novelty candy versus generic food).

Once an explanation is written, GPT-4 switches from explainer to simulator. Given only the sentence and a new passage, GPT-4 predicts neuron activations on the same 0-10 scale. Two simulation regimes were tested. One-at-a-time simulation queries GPT-4 separately per token, which is accurate and expensive. All-at-once simulation marks unknown activation slots in the full sequence and fills them in one pass; it is faster and, in the paper’s comparison, comparable or slightly better, which is what makes scanning all the neurons in a wide MLP block feasible. A weather-phenomena caption, for example, should assign high simulated activations to thunderstorm, hail, and lightning and near-zero to function words. Those simulated activations are the object that scoring explanations actually grades. GPT-4 as simulator is therefore not a critic of prose quality; it is a stand-in for the hypothesis “if this sentence were the mechanism, where would the neuron fire?” Evaluator circularity is real: the same model family both authors and grades the explanation, so shared biases can inflate explanation scores without matching a human concept. The paper’s defense is empirical scoring against the subject neuron, not against another GPT-4 caption. Still, circularity belongs in any honest methods section. A smart human looking at the same traces would also overfit small excerpt sets; GPT-4 simply does it at batch scale. Simulation quality also inherits GPT-4’s known weaknesses on exact position and counting, which is why constructed puzzles that depend on “fifth token in the sentence” are hard even when the verbal rule is simple.

rho = corr(a_real, a_sim)   # Pearson correlation after linear calibration of simulated activations

Scoring explanations uses correlation between simulated activations and actual neuron activations across many tokens. Before scoring, simulated activations are linearly calibrated to the real activation scale so the metric asks whether GPT-4 ranked the right tokens high, not whether it guessed raw magnitudes. A score near 1.0 would mean the explanation fully captured the firing pattern; 0.0 means no linear relationship; negative scores mean the caption predicts the opposite of the neuron. Two regimes matter. Top-and-random scoring mixes high-activating excerpts with random excerpts and tests whether the explanation distinguishes the neuron’s preferred contexts from typical text. Random-only scoring uses only random text and is stricter: it penalizes captions that fit a few dramatic fires and miss broader behavior. Explanation scores therefore measure predictive constraint, not literary quality. Correlation is used rather than explained variance because activations are skewed and non-Gaussian. Unigram baselines complicate any celebration of GPT-4. A lookup table of token-averaged neuron activations, or a linear dot product between token embeddings and the neuron’s input weights, can outperform a one-sentence GPT-4 caption on raw prediction because natural language is a compressed gist. Even a smart human could not pack every token quirk into one clause. Natural-language explanations still supply an interesting qualitative understanding that tables do not: a researcher can read “legal proceedings” and design an intervention, whereas 50,000 token means are opaque. Hybrid pipelines that ask GPT-4 to summarize a lookup table try to keep both. Token-based prediction from weights works relatively well in early layers, where neuron activations are more lexical; GPT-4 explanations overtake that baseline in deeper layers where context dominates. Including positional embeddings in the linear baseline yields only minor gains. Lookup tables typically beat weight-only prediction and sometimes beat explanation based prediction in correlation, while remaining unreadable. The paper also built synthetic neuron puzzles with known ground-truth rules – historical years, token position, conditional triggers – to test the loop with an answer key. GPT-4 as explainer solved only a minority under the baseline prompt; revision recovered additional puzzles; the scorer ranked the true explanation first or near-first in nearly all puzzles. That pattern is the methods moral: scoring explanations is ahead of generating them. The bottleneck is hypothesis search from limited excerpts, not the inability to recognize a good caption when one is supplied. Hyperparameters still move the distribution: context length, GPT-4 temperature, few-shot exemplars, number of top spans, and the mix of random versus top evaluation tokens can all shift explanation scores. More top spans clarify the pattern and risk overfitting; more random spans test generalization and can drown sparse units. None of these knobs turns a polysemantic unit into a monosemantic sentence.

• Top-and-random versus random-only: the former is the headline scoring explanations regime; the latter is the stricter test of whether captions generalize off the peak fires.
• Weight baseline: embedding dot product with the neuron input vector, ignoring context and nonlinearity; strong in layer 1, weak where neuron behavior is contextual.
• Lookup-table baseline: per-token mean neuron activations; high predictive power, very low human interpretability.
• Hybrid: GPT-4 reads table statistics and writes a summary used as the simulator prompt, trading some accuracy for an interesting qualitative understanding.
• Puzzles: ground-truth synthetic neurons show that GPT-4’s explainer, not the correlation scorer, is the main failure point, especially on positional and counting rules.

Why Researchers Explain Neurons in GPT-2 XL’s 307,200 MLP Units

GPT-2 XL was the testbed because its weights are public, its width is large enough to be interesting and small enough to scan, and its architecture numbers are independently documented. The 2019 GPT-2 report specifies 48 transformer layers. OpenAI’s 2023 explainer paper then treats 6,400 MLP units per layer, so the MLP corpus is 307,200 neurons (48 x 6,400) in a 1.5B-parameter network. That is the scale at which manual inspection fails and at which language models become plausible annotators. The pipeline processes all the neurons using tens of millions of neuron-token activation pairs. Subject models also include GPT-2 Small and GPT-3 variants so the authors could ask whether larger subjects are harder to explain, but GPT-2 XL is the public, full-width MLP scan. MLP neurons were chosen rather than attention heads, residual-stream directions, or full circuits because they are the simplest indexed nonlinearity where large-scale automation is feasible. Attention routing, multi-neuron circuits, and long-range dependencies are deliberately out of scope. From a Jacobian Lens view, each unit is one coordinate in a large Jacobian: not the most natural axis, but an axis the model already has. The analogy to single-cell recording is limited and should stay limited. Isolated tuning curves can motivate circuit work; they do not replace it. Language models can serve as computational probes of other language models in the same modest sense.

A “neuron” here is not a biological cell and not automatically a feature. In GPT-2, each layer’s MLP applies a linear map, a GELU, and a second linear map that writes back to the residual stream. For each token, each unit emits a scalar: that scalar is the neuron activation. Those neuron activations are the time series the explainer sees and the simulator must match. Features, by contrast, are meaningful directions in activation space – dates, legal register, sarcasm – which may or may not align with individual neurons. In an ideal basis each unit would be monosemantic. In trained language models, superposition packs more features than units by using approximately orthogonal high-dimensional directions, so many neurons are polysemantic. A unit may fire for programming syntax and French cuisine without those concepts being related. That is why later work treats sparse autoencoder latents as candidate features and why this paper’s choice to interpret neurons should be read as a coordinate system, not as ontology. Connection weights make the coordinate concrete. A neuron’s input weight is its row in the MLP input matrix, composed with preceding layer-norm; the token-to-neuron influence in the linear approximation is a dot product between the residual representation and that weight vector. The output column writes the activation back to residual space and, after later norms, toward vocabulary logits. Inter-layer neuron-to-neuron influence is a product of output and input weights mediated by attention and norm, which is the path Jacobian Lens methods formalize when they ask how a perturbation at one unit moves activations elsewhere. If the embedding for Thor aligns with a unit’s input weight, that unit’s neuron activations rise on Thor; that fact is still only a local linear story, not a circuit. Different activation functions change how readable those scalars are. Sparse ReLU-like fires are easier to caption than dense GELU fires because many tokens are cleanly zero; GPT-2 uses GELU, which is part of why a large fraction of units resist one-line descriptions. Models using different activation functions in ablation settings showed different explainability profiles: sparsity helps captions and can cost parameter efficiency. Training duration also moves the picture. Longer training can sharpen top-and-random explanation scores while hurting random-only scores if more features share each neuron. None of that requires treating GPT-2 neurons as brain cells. Research on language models and neuroscience can share computational metaphors only if the disanalogies stay explicit.

• Polysemantic neuron: one unit, several unrelated concepts, so a single English phrase cannot be a complete mechanism.
• Feature direction: a linear combination of neurons that more cleanly tracks one concept, the object later SAE work tries to isolate.
• Activation asymmetry: many neurons fire rarely and weakly, then spike on a narrow set, which is why top-activating excerpts dominate prompts.
• Superposition: more features than neurons, overlapping directions, the structural reason individual neurons are a messy unit of analysis.

Public datasets follow the same MLP-first choice and come with caveats that affect anyone who wants to interpret neurons from the released files. OpenAI released neuron activations and GPT-4-generated explanations for GPT-2 Small and GPT-2 XL. GPT-2 Small includes MLP neuron activations per token and attention-head activations per token pair, with preprocessing and scoring choices that differ from the XL pipeline. GPT-2 XL releases only MLP neuron activations, covering all the neurons across 48 layers. Because methodologies differ, explanation scores are not directly comparable across those two subjects. An optimized GELU used at GPT-2 XL inference introduced small discrepancies versus the standard implementation; OpenAI acknowledged the issue, and the effect is small but relevant for precise layer comparisons. Activation files live on Azure Blob Storage, linking neuron IDs to per-token neuron activations and to explanation text, which means local reproduction can require cloud credentials. The setting being explained is deliberately narrow: activation magnitude after GELU and before the residual write, correlated with properties of the current token-in-context. High activation means the unit cares about that token-in-context; near-zero means it does not. The caption is supposed to name that pattern. Ignored on purpose: attention-head routing, multi-neuron algorithms, and dependencies that span many layers. That is a practical starting point for language models, not a claim that MLP scalars exhaust model computations. Researchers who want Jacobian Lens or alternative metrics should treat the datasets as a benchmark, not as an unlabeled dump of truth. Cross-model score tables that ignore the GPT-2 Small versus XL protocol split are not evidence. The same caution applies to GPT-3 variant numbers in the paper: they are scaling probes, not a second public neuron browser.

What Explanation Scores Reveal About Neuron Behavior

The headline empirical picture is that GPT-4 can produce higher scoring explanations than smaller explainer models, and that this scaling of the explainer does not imply that most GPT-2 XL units are well explained neurons. Larger GPT-4-class explainers beat weaker GPT-family explainers when the simulator is held fixed, which is the paper’s scaling observation about language models used as annotators. More powerful simulators improve top-and-random explanation scores and plateau on random-only scoring, so even GPT-4 does not turn every unit into a clean detector. Subject-model scaling goes the other way: larger subjects are generally harder. GPT-3 variants are less explainable than GPT-2 under both regimes, especially in later layers, consistent with more entangled neuron behavior as capability rises. Early MLP blocks tend to be more lexical and local; later layers encode more abstract or mixed structure that a single sentence misses. Explanations score poorly for the vast majority of units relative to the hope that auto-captions would saturate the MLP. A long tail of crisp neurons remains: topics, syntax, specific names, date formats, hedging versus certainty. That tail is scientifically useful and easy to over-read. Cherry-picking a comics detector or an HTML-syntax detector is not the result. The result is a score distribution with a thin high end and a bulk of mixed or opaque units. Treating average explanation scores as a verdict on whether transparency is possible would also be a mistake: they measure this pipeline, these prompts, this basis, and this simulator.

Do not treat a fluent GPT-4 sentence as ground truth. The metric is whether simulated activations match real neuron activations, not whether the prose would pass as a feature card. Well explained neurons exist and are worth seeding into circuit work; they are the exception. Random-only scoring is stricter than top-and-random scoring because it withholds the dramatic peak contexts that made the caption easy to write. Layer trends could reflect method limits rather than intrinsic opacity: GPT-4 may lack the concepts, the context window, or the causal access needed for late-layer neuron behavior. Activation-function and training-duration ablations cut the same way. Different activation functions that sparsify fires make captioning easier; longer training can sharpen peak tuning while increasing superposition off-peak. In all of these comparisons, explanation scores are hypotheses about neuron behavior under a frozen natural-language bottleneck. They are not certificates that model computations have been recovered. Any summary that says black-box AI has finally been explained is describing a different paper from the one OpenAI ran.

Polysemanticity, Interesting Neurons, and Marvel Cinematic Universe Detectors

Polysemanticity is why auto-captions hit a ceiling, and it is the main scientific obstacle rather than UI polish. If one neuron mixes unrelated concepts, a single English phrase cannot be a complete mechanism, and the technique works poorly exactly where safety-relevant structure may be distributed. Superposition is the hypothesized reason: language models encode more features than they have neurons by overlaying directions. Anthropic’s 2023 monosemanticity / dictionary-learning result independently stressed the same limitation the same year: many individual neurons are polysemantic, and learned features are often more interpretable than raw neurons. Failure modes follow. GPT-4 can confabulate a tidy label that fits the five peak spans and misses the rest. Dataset-specific artifacts (token splits, boilerplate, a site’s house style) can look like concepts. Explainer-simulator circularity can reward GPT-4-typical glosses. Confabulated explanations still receive a number; that number can be low, which is the point of scoring, but a middling score plus a fluent sentence is easy to over-trust in slides. The technique works poorly on highly polysemantic units, and those units may matter more for auditing than the well explained neurons in the long tail. Looking beyond individual neurons is therefore not an optional sequel. The 2023 paper already tries covariance-aware reparameterization and coordinate ascent to find directions whose activations GPT-4 can explain better than single-unit traces. Optimized directions often earn higher scoring explanations than random neurons, and some directions admit labels no single unit captures. Recovering them can require dozens or hundreds of basis components, which is what superposition predicts. From a J-Space perspective, directions are closer to broadcast-like variables than neuron indices are, but they are still correlational until someone patches them. Finding explainable directions does not dissolve polysemanticity in the original basis; it changes the unit of analysis, which is the move SAE autointerp later made systematic.

The curated lists of interesting neurons are still worth reading if they are treated as entry points, not as the finding. Public examples include programming-syntax units (brackets, indentation, definitions), date and temporal units, certainty versus hedging units, simile-structure units that prefer literary “X is like Y” over technical comparison, and several units selective for Marvel comics and Marvel Cinematic Universe text. Marvel Cinematic Universe neurons fire on character names, plot summaries, and genre vocabulary; one striking pattern is accidental heroic destruction, as when Thor wrecks a building, as opposed to deliberate villainy. Those units also respond to ice and flame beams, magic power rings, and story beats about earth bound villains. Passages of heroes facing earth bound villains, or facing earth bound villains who are framed as saving Earth from a cosmic threat, or an embarrassing heroic loss, produce distinctive neuron activations. Ice and flame beams and magic power rings show up as genre furniture rather than as a token whitelist; the claim in the case studies is that some units track narrative tone and trope, not only names. Adjacent entertainment-coverage units fire on Night Wing movie chatter, Dick Grayson, or Matt Reeves. Tokenization artifacts sit inside the same lists: St Ark fragments from Stark, and “weren t” from weren’t, so a neural network may lock onto BPE pieces that have no standing as concepts. Kit Kat units illustrate the opposite extreme of specificity: not generic candy, but novelty Japanese product copy, just a new flavor entering that market. Just a new flavor is both a product pattern and a reminder that pretraining memorizes fine clusters. Simile and rhetoric neurons, certainty neurons, and these pop-culture units collectively show that language models carve training data at a granular level. They do not show that the model “thinks in Marvel.” Scientifically, the question is whether such interesting neurons are specialized detectors or partial coordinates of a larger subspace. That is why they are good testbeds for path patching, Jacobian Lens mapping, or workspace-style probing, and a bad basis for claiming that to explain neurons is to explain the network. Lists are filtered by explanation scores, activation variance, and token association. They help humans start; they do not replace scoring on held-out neuron activations. If the only figures a reader remembers are comics, citations, and HTML detectors, that reader has substituted the gallery for the distribution. Complex behavior in late layers, mixed concepts on one unit, and low explanation scores on the bulk of MLP width remain the typical case. Complex patterns that span registers will not fit a caption even when GPT-4 is willing to write one. Interesting qualitative understanding is real for the tail and insufficient for the typical unit. Researchers should inspect top activating examples, simulated versus real traces, the score, and counterexamples before promoting any unit to a named feature.

• Syntax and time: programming punctuation and date formats are among the more captionable interesting neurons because the support in text is local and repetitive.
• Rhetoric: simile detectors and certainty/hedging detectors show neuron behavior that is structural rather than topical.
• Marvel Cinematic Universe: multiple interesting neurons track superhero narrative, including ice and flame beams, magic power rings, earth bound villains, and facing earth bound villains as a plot template, plus collateral-damage-by-hero tropes.
• Marvel Cinematic Universe coverage versus comics: entertainment-press spans (Night Wing, Grayson, Reeves) can drive neuron activations as strongly as in-universe prose, which is a data artifact as much as a “comics feature.”
• Consumer clusters: Kit Kat novelty copy is just a new flavor of the same phenomenon: narrow thematic clusters that GPT-4 can name and that still may be polysemantic off-peak.
• Tokenization: St Ark and similar splits mean some interesting neurons are detectors of BPE accidents; explanations that ignore that fact will not generalize.
• Directions: linear combinations can yield higher scoring explanations than individual neurons, at the cost of a less convenient basis than the MLP index.

How to Interpret Neurons With Public Code and the Neuron Browser

Practitioners should start from the two public surfaces OpenAI actually shipped, not from screenshots of cute units. The interactive HTML paper and neuron browser is for inspection: explanations, top activating examples, and navigation across GPT-2 XL MLP width. The automated-interpretability repository is for pipelines: compute neuron activations, format explainer prompts, call GPT-4 (or a substitute) to generate explanations, run the simulator, and score. Connecting claims about neuron behavior to those scripts is the difference between citing a vibe and citing a method. A useful inspection order is: top activating spans, the frozen caption, simulated activations versus real neuron activations, the explanation score, then deliberate counterexamples the caption should reject. If simulated activations match the peaks but also fire on clearly irrelevant tokens, the unit is a candidate polysemantic or the caption is too broad. If the caption is specific and the score is still weak, GPT-4 may have missed the true pattern, or the pattern may not be verbalizable. Reproducing a handful of neurons is feasible for a lab that can run GPT-2 XL forward passes and pay for GPT-4 (or an open explainer). Re-explaining all the neurons is a research-scale job: GPT-2 XL alone is 307,200 MLP units, each needing excerpts, an explainer call, simulator calls, and scoring. The methodology’s long-term ambition is to analyze millions of neurons for audit use; current costs make exhaustive frontier-model scans impractical without heavy subsampling. Instruction-tuned fine tuning variants, multilingual models, and open weights such as LLaMA or Mistral families are natural replication targets precisely because full-width GPT-4-subject scans are not. Using smaller open language models as explainer/simulator trades peak explanation scores for inspectability and cost. Swapping Pearson for rank correlation, mutual information, or ablation-based causal scores is a concrete next experiment, not a slogan. Neuron explanations can seed path patching: a “legal terms” unit becomes a handle for asking which heads route legal context into that MLP. Jacobian Lens analysis can then ask how that unit moves downstream logits. None of those follow-ups is bundled in the 2023 repo; the repo is the hypothesis generator plus scorer.

• Activation extraction: scripts compute per-token neuron activations for specified MLP layers on a corpus.
• Excerpt packing: tools select top activating windows and format them for GPT-4, including the 0-10 annotation scheme.
• Prompt templates: separate templates to generate explanations and to simulate activations, plus scoring code for explanation scores.
• Neuron viewer: browse curated interesting neurons, captions, and examples without re-running the full loop.
• Storage: datasets on Azure Blob Storage; early docs noted credential pitfalls for local reproduction.
• Practitioner checklist: real trace, GPT-4 caption, simulated activations, score, counterexample, only then any safety story.
• Compute: explaining neuron behavior across GPT-2 XL implies a very large number of GPT-4 calls; context length and cost scale together.
• Tokenization: BPE splits distort both explanations and simulations; GPT-4 may not know how a unit should treat an arbitrary subword.
• Context window: long-range neuron behavior that needs hundreds of tokens will not fit a short explainer context; longer windows help and cost more.
• Fine tuning: applying the same loop after instruction fine tuning is an open empirical question, not a documented 2023 result, and should be tagged as such.

Practical constraints are part of the method, not footnotes. Explaining neuron behavior is compute-heavy even though interpretability compute is still smaller than pretraining. GPT-4 as explainer and simulator across GPT-2 XL’s 307,200 neurons is already an industrial loop; a GPT-4-class subject would be worse. Context-length limits bind when neuron activations depend on long-range structure the explainer cannot see. Tokenization issues are systematic: explanations that mention nonsense fragments, simulations that fail on subwords, and bias against rare morphology. Jspace.com treats tokenization as part of the Jacobian Lens basis: the token alphabet changes which neurons look simple. Alternative tokenizations or continuous-input models would change the explanation scores without changing the underlying computation. Community infrastructure still missing includes larger puzzle suites than the paper’s synthetic set, shared leaderboards for explainer and simulator variants, and common datasets on modern architectures. Those are well-posed research programs: specified inputs, outputs, and metrics. They are how to build on Bills et al. without recycling circuits rhetoric that skips explain-simulate-score. Researchers can also test whether explanation based labels transfer across languages, whether open language models can replace GPT-4 in either role, and whether causal scores overturn high Pearson explanation scores. Until those measurements exist, the public browser is a hypothesis catalog. It is not completely pointless, and it is not a completed map of model computations. Dismissing the artifacts as completely pointless ignores the filtering function: low-cost captions plus a falsifiable activation-match test, with code on GitHub, is how the field got a baseline that later SAE autointerp could cite instead of starting from anecdotes.

Sparse Features, Autointerp, and Language Models After the 2023 Baseline

What this paper still uniquely teaches in 2026 is cheap natural-language hypotheses plus a falsifiable activation-match test, not that interpretability was solved in 2023. After the neuron-captioning baseline, dictionary learning and sparse autoencoders changed the unit of analysis. Anthropic’s monosemanticity work argued that features recovered by dictionary learning are often more interpretable than raw neurons, which is the complementary result to OpenAI’s finding that a vast majority of MLP units resist a single caption. OpenAI and others later applied auto-interpretation to SAE features rather than only to raw neurons, keeping the same outer loop: generate explanations, simulate activations, score. That continuity is the honest citation. Bills et al. 2023 is the obvious baseline method for autointerp, including when the vectors being labeled are latents rather than GELU units. It is not a proof that large language models are interpretable, and it is not obsolete. Feature-level autointerp still inherits confabulation risk, explainer-simulator circularity, and the temptation to read a fluent GPT-4 sentence as a mechanism. The difference is that the basis is chosen for sparsity and monosemanticity rather than for being the MLP index. Researchers who skip the 2023 neuron paper and jump to SAE dashboards lose the control experiment: if GPT-4 cannot stably caption even the cleanest GPT-2 XL units, feature captions need the same scoring discipline. If GPT-4 captions features more easily than neurons, that is evidence for the basis change, which is a comparative claim this 2023 corpus makes possible. Scaling trends from the original work still frame expectations. GPT-4 as a stronger explainer helped; larger subject models hurt; later layers hurt; different activation functions that sparsify helped. SAE features are, in part, an attempt to manufacture the sparsity that made some neurons captionable. That is a mechanistic bet, not a marketing upgrade.

Cite the paper as a method paper on automated hypothesis generation and scoring. Correct the viral error that GPT-4 explained its own neurons. The target was GPT-2 XL MLP neurons; the explainer was GPT-4; the date was May 9, 2023; the width was 307,200 units across 48 layers. Use it when you need a reference for scoring explanations against neuron activations or feature activations. Do not use it as evidence that language models can be certified, that alignment is solved, or that a Marvel Cinematic Universe detector explains planning. Safety relevance is scalable oversight of internals: a way to propose what a coordinate might do and to fail the proposal when simulated activations do not match. It is not a substitute for behavioral evaluations, red-teaming, or governance. The NIST AI Risk Management Framework treats explainability and interpretability as trustworthy-AI characteristics; this lab pipeline is one research instrument often cited against that frame, not a compliance artifact. In principle, teams could search for neurons or directions whose GPT-4 captions mention deception, manipulation, or policy-violating content and then review those components. In practice, current methods remain incomplete. Distributed features will not show up as a single well explained neuron. Larger models tend to yield weaker explanation scores, which makes automated screening harder where it is most requested. Neuron explanations should remain one diagnostic among many. Alignment implications are therefore modest and specific: filtering, triage, and hypothesis generation for mechanistic follow-up. Global-workspace-style stories should stay equally modest. Consistently high-activation directions that affect many layers could be candidates for broadcast-like signals in a transformer; current captions do not establish subjective experience, biological equivalence, or consciousness. Jspace.com is interested in testing whether language models show any workspace-like dynamics; that remains an open empirical question. Treating high-scoring GPT-4 blurbs as inner speech would be a category error.

Limitations remain the right closer. The pipeline explains correlations between tokens and neuron activations, not the mechanisms behind neuron behavior. Knowing that a unit fires on legal terms does not identify the heads that routed that context or the causal effect on logits. Simulator collusion, a narrow hypothesis space restricted to single-neuron token-local patterns, and polysemanticity are structural. Not all neurons need to be individually interpretable; finding readable neuron activations does not mean the neural network “thinks” in those labels. The appropriate stance is to treat captions as hypotheses for ablation and tracing. Dismissing the approach as completely pointless would still be wrong: imperfect explanations at scale filter the MLP for units worth expensive tools, and the open benchmark lets others raise explanation scores with better explainers, better bases, and better metrics. Better explanations from revision, from GPT-4 scaling, and from direction search are all documented as incremental, not as saturation. Automatically write explanations, score them, publish the misses. That loop is the lasting artifact. Combining it with causal interventions, circuit tracing, and SAE features is the path that preserves what the 2023 paper actually showed about language models, GPT-4, and the difficulty of naming neuron behavior without pretending the work finished the job. Readers should use the original research page, the HTML paper, and the GitHub repository as primary sources, keep the GPT-4 / GPT-2 XL division of labor explicit in every citation, and demand simulated activations whenever a caption is offered as evidence about model computations.

Editorial Team

Editorial Team

The J-Space Editorial Team curates and synthesizes research in mechanistic interpretability, AI alignment, and cognitive science. With a commitment to precision, empirical rigor, and clear exposition, the team distills complex technical developments into accessible yet authoritative analyses for researchers, engineers, and policymakers working at the frontier of artificial intelligence.