Sparse Autoencoders Use L1 Penalties to Isolate Distinct Model Features

Updated August 10, 2026

Modern language models represent many overlapping concepts in dense residual-stream vectors. A single neuron or residual-stream dimension may activate for both “French text” and “Python code comments,” making neuron-level interpretations unreliable. Polysemanticity complicates neural network interpretability because there is no clean one-to-one mapping between internal units and human concepts. Sparse autoencoders aim to disentangle polysemantic activations by projecting these dense vectors into a larger, sparser coordinate system where each active coordinate fires for one concept. Papers like “Sparse Autoencoders Find Highly Interpretable Features in Language Models” (Cunningham et al., 2023) and Anthropic’s scaling reports (2024) popularized this approach. This is how sparse autoencoders find features in language models.

This article clarifies the full pipeline from collecting language model activations, training the SAE, the geometric process of feature discovery, the issue of composed features that occur frequently, and methods for validation such as steering. It does not claim that SAEs solve all interpretability problems or provide direct insight into consciousness without further causal evidence.

The Challenge of Polysemanticity in Neural Networks

Modern language models represent many overlapping concepts in dense residual-stream vectors. A single neuron or residual-stream dimension may activate for both “French text” and “Python code comments,” making neuron-level interpretations unreliable. Polysemanticity complicates neural network interpretability because there is no clean one-to-one mapping between internal units and human concepts. Sparse autoencoders aim to disentangle polysemantic activations by projecting these dense vectors into a larger, sparser coordinate system where each active coordinate fires for one concept.

Polysemanticity arises from superposition in neural networks. There are more abstract features a model needs to represent (all concepts from an internet-scale corpus) than there are neurons, so many features share neurons, creating polysemantic units that respond to multiple unrelated concepts. This is the role in understanding superposition in neural networks.

Anthropic’s 2022 “Toy Models of Superposition” (Elhage et al.) formalized this as a geometric problem: features are sparse but stored in a lower-dimensional space, forcing them to overlap. The model can store features in non-orthogonal directions within a lower-dimensional activation space, producing mixed activations where any single neuron’s firing pattern encodes multiple features simultaneously. This same antipodal feature storage geometry appears in both toy models and real neural network internals.

SAEs address this by mapping dense activations (e.g., 4,096 dimensions in Pythia-70M) into a higher-dimensional sparse feature basis (e.g., 16,384 latent units) where each unit should fire for one interpretable feature direction. The remaining sections show how this mapping is learned, under what conditions it recovers true underlying features, and when it settles on composed alternatives. This is how sparse autoencoders find features in language models.

Core Architecture of Sparse Autoencoders

A sparse autoencoder is a two-layer neural network (encoder + decoder) trained to reconstruct its input activations while forcing the hidden layer to be sparse. SAEs are powerful tools for mechanistic interpretability and can discover hidden properties without manual labels. The concrete architecture used in language model work: the encoder is a linear map plus ReLU (or a variant like BatchTopK), the decoder is a linear map back to the original activation dimension, and the hidden dimension size is often much larger than the input.

A sparse autoencoder’s hidden layer is often much larger than the input activation vector. In Cunningham et al.’s Pythia-70M experiments, the input dimension was 4,096 and the latent width was 16,384. Anthropic’s Claude 3 Sonnet work used up to 34 million feature decoder vectors.

The training objective: SAEs are trained to minimize reconstruction loss and sparsity penalties. The reconstruction term is mean squared error between the original activation and its reconstruction; the sparsity term is typically an L1 norm on hidden activations. SAEs use a sparsity penalty to encourage fewer active neurons, achieving monosemanticity by activating fewer latent neurons per input. The encoder and decoder weights in sparse autoencoders are often constrained to prevent feature collapse; for example, decoder column normalization prevents trivial rescaling where the encoder shrinks activations and the decoder inflates them, cheating the sparsity penalty.

Loss = MSE(x, decoder(encoder(x))) + λ * ||encoder(x)||_1   # x is input activation vector from language models; MSE measures reconstruction; λ weights the sparsity term on hidden activations

The training process of sparse autoencoders involves a forward pass for encoding and a reconstruction for decoding. Once trained, the columns of the decoder matrix are treated as candidate feature directions in the original model’s activation space, each one a feature vector that the SAE has learned to detect.

Training Process for Sparse Autoencoders on Language Models

The pipeline from raw text to trained SAE follows these steps. Collect activations. Run a pretrained transformer language model on a large dataset (Common Crawl, Wikipedia, or curated corpora). Record residual-stream vectors from chosen layers. Each activation is a high-dimensional vector; for a 7B-parameter model this might be 4,096 dimensions per token.

Build the training set. The training data distribution should contain tens or hundreds of millions of token-level vectors to capture diverse linguistic contexts. In Anthropic’s Claude 3 Sonnet work, the activation dataset was drawn from a text mixture similar to the model’s pretraining data.

Train the SAE. Activations are batched into the SAE. The encoder produces sparse hidden vectors; the decoder reconstructs activations. Gradients update both matrices to reduce reconstruction error while enforcing sparsity. SAEs learn to represent activations as sparse feature combinations; only a few features fire per token. SAEs learn to represent model activations as sparse combinations of features.

Sweep hyperparameters. Researchers sweep latent width, sparsity coefficient, and learning rate layer by layer. Anthropic’s scaling laws showed loss decreasing as a power law in compute, with optimal feature count increasing faster than training steps. Their 34-million-feature SAE explained at least 65% of activation variance, with fewer than 300 features active per token on average. SAEs transform input vectors into larger, sparser representations. An SAE may split one broad feature into several more specific features during training as the latent width increases. This step-by-step training of a sparse autoencoder allows systematic exploration of feature recovery from language model activations.

How Sparse Autoencoders Find Features

The SAE learns a new linear basis for the activation manifold in which most coordinates are zero, and active coordinates trace out sharp directions corresponding to features. From a Jacobian lens perspective, this reparameterizes the local geometry so that each feature direction discovered by the decoder aligns with a recognizable concept. How sparse autoencoders find features involves optimizing for both reconstruction and sparsity to isolate these directions in activation space.

After training, each column of the decoder matrix is a candidate feature vector in the original residual-stream space, while the corresponding row of the encoder detects when a feature vector input projects onto that direction. Sparsity makes feature activations rare but strong: on typical tokens only a handful of latent units fire, approximating classical sparse coding (Olshausen & Field, 1996).

“Finding” a feature means that whenever a certain concept appears in context (“prime numbers,” “Spanish cities,” “markdown bullet list”), one particular latent unit tends to fire, and its feature direction causally influences the model’s behavior when manipulated. Interpretability in SAEs arises from optimizing for sparsity and reconstruction simultaneously, forcing the SAE to pick out the highest-value directions. But the mapping is not perfect. Because the input data itself contains composed patterns, the SAE may allocate units to joint configurations rather than single atomic concepts. SAEs help disentangle polysemantic activations into monosemantic vectors, but the degree of disentanglement depends on the feature distribution in training data and the sparsity regime.

The Problem of Composed Features

A “true” feature corresponds to a single underlying feature, while a composed feature corresponds to a recurring combination of underlying features. Consider a latent that fires only when a token is inside both a “French text” span and a “city name” span (e.g., “Paris,” “Lyon”). This composed feature pair encodes “French city,” even though “French” and “city” are separable concepts. Polysemanticity makes it hard to evaluate learned features’ robustness in part because composed features can look just as interpretable as atomic ones. Synthetic work from 2022 through 2025 shows the mechanism: when two features almost always appear together, the SAE prefers a single composed latent over two separate latents to minimize loss under a fixed sparsity budget. If composed pairs occur at high frequency and individual features occur only rarely, the SAE has little incentive to devote separate capacity to each atomic direction. Researchers create synthetic data to prescribe ground-truth features and measure when SAEs recover them versus learning composed features. These synthetic data vectors stand in for language model activations with structured superposition. A typical construction builds an extremely simple toy model with four true underlying features arranged in a 2×2 grid. Each sample can activate features in defined combinations. By controlling the example probability table of pairs (e.g., how often each of the four possible feature configurations appears), we adjust how often composed features appear and how feature amplitude correlation affects outcomes. Some experiments also test perfectly correlated feature amplitudes versus perfectly uncorrelated feature amplitudes to isolate the effect of correlation on composed feature learning. Some setups introduce one-hot vectors (only a single true underlying feature active) with probability p (e.g., p = 0.25) to give the SAE a chance to see isolated true features rather than only pure composed pairs. The input data vectors are structured so researchers can measure recovery of independent feature sets versus composed feature sets. These sample composed data vectors let us trace exactly when SAEs faithfully recover underlying bases and when they settle on entangled codes. The data vectors composed from known atomic signals provide a controlled testbed for every architectural variant. Even when true features are known and simple, SAEs frequently converge on latent units aligned with composed pairs when those pairs dominate the training data distribution. The trade-off is that given a fixed sparsity penalty and latent width, allocating separate latents for two tightly correlated features costs more sparsity (two active units) than using a single composed latent (one active unit), while reconstruction loss barely changes. When a model learns composed pairs, it achieves lower total loss. A composed feature pair giving lower total cost than two separate entries is a direct consequence of L1 geometry. When feature amplitudes are perfectly correlated across dimensions, SAEs learn off-axis composed feature directions echoing antipodal feature storage geometry. The middle and right panels of typical toy-model visualizations show feature decoder vectors collapsed onto composed directions rather than aligned with ground-truth axes. Even when either the same solution or a decomposed solution exists, the optimizer gravitates toward the composed feature set. Varying hyperparameters (learning rate, sparsity coefficient, decoder normalization) shifts the degree but often does not eliminate learning composed features. When least frequently occurring features have low probability, the SAE ignores them. This bias matters: if SAEs find composed features even in toy settings, interpretability claims about specific features in full language models must be tempered by awareness of this pattern. The feature distribution affects what the SAE can recover; the rest of this article covers diagnostics for telling composed features apart from genuine atomic features. Composed features occur frequently in both toy and real settings.

Interpreting and Steering with SAE Features

Once SAEs are trained on real language model activations, we must identify what each latent unit represents. Training SAEs can help disentangle polysemantic activations, but labeling is where the rubber meets the road. Autolabeling pipelines collect, for each latent index, the top-activating tokens, prompts, or contexts, then summarize them with a short natural-language label such as “Latin script URLs,” “XML tag boundaries,” or “polite refusal style.” Researchers present ranked token lists, example sequences, and activation heatmaps so they can visually judge whether SAE features reveal interpretable features or noise.

Proxy metrics include variance explained (FVE), L0 sparsity, and auto-interpretation score. Real examples of features like specific concepts in LLMs include directions that correspond to “markdown list,” “negative sentiment,” or “code errors” rather than opaque individual neuron activations.

Ma et al. (2026) tested whether features labeled as “reasoning” actually corresponded to reasoning operations. Across 20 configurations, 59% to 94% of such features could be activated by injecting lexically similar tokens into non-reasoning text. This falsification approach is critical: because composed features can still score as interpretable (e.g., “French city names”), interpretability evaluations must account for the composed-feature caveat, especially for circuits and global workspace arguments about internal “thoughts.” Sparse autoencoders find human-interpretable features in language models, but the interpretability of those features needs careful testing.

Once we have model’s feature representations as decoder vectors corresponding to meaningful features, we can intervene on model activations by adding or subtracting those directions and observe how outputs change. The basic steering procedure: encode an activation with the SAE, tweak one latent coordinate up or down, decode back to the residual stream, and feed that modified activation forward through the language model. In Anthropic’s Claude 3 Sonnet work, amplifying a “Golden Gate Bridge” feature caused the model to identify itself as the Golden Gate Bridge under high activation. Suppressing a “code errors” feature shifted code-generation behavior. These are not composed pairs producing ambiguous results; they are cases where a feature occurs in isolation and steering confirms its role. Successful steering experiments provide causal evidence that a latent corresponds to a real, behaviorally relevant feature. But composed features can still steer joint patterns (e.g., “first-person apology in English”) rather than atomic concepts. The fewer features you manipulate at once, the cleaner the causal test. Comparison to other interpretability methods like probing shows that SAEs require no labels unlike probing but may learn composed features from language model activations.

FAQ

How do sparse autoencoders differ from standard autoencoders in practice?

Standard autoencoders compress to a smaller dense bottleneck and optimize only for reconstruction error. SAEs for language models do the opposite: they expand to a larger latent space with explicit sparsity constraints so that only a few coordinates are active per token. This means the hidden dimension is larger than the input, not smaller. The sparsity penalty forces each latent to specialize, producing features occurring in narrow contexts rather than dense encodings that blend everything together.

Can SAEs prove that a language model represents a specific human concept?

SAEs provide suggestive evidence by finding directions whose feature activations correlate with human-labeled patterns and whose manipulation causally steers outputs. They do not mathematically prove that internal states match human concepts one-to-one. A latent unit that fires for “sarcasm” may capture a statistical correlate of sarcasm rather than the model’s internal representation of the concept. Features occurring in correlated contexts may be composed of simpler underlying true features that the SAE failed to separate.

Why do researchers care so much about composed features if they still look interpretable?

For safety and theory-building, we need to know whether a feature corresponds to a single abstraction or a correlation bundle. A composed feature that blends “refusal” and “English” hides substructure that matters for understanding reasoning circuits or possible global workspaces. If you try to suppress “deceptive behavior” using a feature that is actually a composed feature pair of “deception + formal register,” you may accidentally suppress formal responses that have nothing to do with deception. Individual features matter because they are the atomic units of causal analysis.

Are there alternatives to SAEs for understanding language model internals?

Alternatives include sparse probing (training linear classifiers on activations for specific tasks), dictionary learning directly on gradients or Jacobians, direct circuit tracing of attention heads and MLP blocks, and other representation learning methods such as Independent Component Analysis. SAEs are one tool in a broader mechanistic toolkit. Each method has different trade-offs: probing requires labeled data for each concept, circuit tracing requires manual hypothesis formation, and SAEs risk learning composed features but require no labels.

Do SAEs scale to frontier models, or are they only for small research systems?

Recent work (2023 through 2026) has trained SAEs on mid-size and large language models by sampling activations from selected layers and tokens. Anthropic trained SAEs with 34 million features on Claude 3 Sonnet’s middle residual stream layer, demonstrating feasibility at production scale. The computational cost is substantial; Anthropic described needing to shuffle terabytes of data with distributed systems. Full coverage of all layers and tokens in trillion-parameter models remains an open engineering challenge, but selected-layer SAEs on frontier models are already producing useful learned features for interpretability research.

How does the sparsity penalty promote monosemantic features over polysemantic ones?

The L1 penalty in the loss function increases the cost of activating multiple latents, incentivizing the SAE to assign each latent to a single coherent concept rather than mixtures. This encourages the discovery of feature directions that are selective for specific patterns in the input data from language models, though it does not fully eliminate composed feature sets when correlations are strong in the training data distribution.