Why Extra Residual Features Fail to Add Workspace Slots

Fact-checked by the J-Space editorial team

Updated August 23, 2026

The residual stream is the only all-to-all communication channel in decoder-only language models, so residual stream width is the right architectural bound on global workspace capacity claims. In the Anthropic Transformer Circuits framework, every attention block and MLP reads that shared vector and writes an additive update back into it; there is no second bus.

This article clarifies what a width-indexed capacity claim can and cannot say. It does not treat global workspace theory as a certificate of phenomenal consciousness, and it does not invent a new empirical map of residual stream geometry. It uses published hidden sizes and superposition geometry, together with sparse-autoencoder evidence of packing, to separate a limited-capacity broadcast hypothesis from an unbounded residual metaphor. A weaker reading remains available: the residual stream might still implement some of the coordination functions people want from a workspace. The open question is how much simultaneous occupancy that implementation can support.

Why a shared residual vector is not a workspace

A plain definition is required before any global workspace analogy can be disciplined. In a decoder-only block the residual stream is the running hidden state attached to each token: a vector in R^{d_model} that is initialized from the token embedding plus position information and then updated, in series, by every attention block and every MLP along network depth. The config field usually called hidden size, model dimension, or hidden dimension is this width. It is not the parameter count, and it is not the number of layers. Residual stream width is the only all-to-all vector that every submodule can read and that every submodule can write, and transformer residual streams therefore function as a shared bus rather than as a set of named registers. Large language models differ in depth and in training recipe, but they do not escape this fact: if a later layer is to use a feature, that feature must already be present or reconstructible from the residual stream at that site. The residual stream is token-indexed. Unless the architecture adds recurrence or an external memory that is not itself implemented through attention writes into the same residual stream, there is no second workspace sitting beside the bus.

x^{(l+1)} = x^{(l)} + A^{(l)}(x^{(l)}) + M^{(l)}(x^{(l)})
# x^{(l)} in R^{d_model} is the residual stream after layer l
# A^{(l)} is the attention write; M^{(l)} is the MLP write

That additive update rule is the opposite of a set of addressable, independently updatable slots. A slot workspace would let a controller write item k without rotating or overwriting item j. Residual addition does not do that. Each write is a vector in the same R^{d_model}, and later writes are free to cancel, rotate, or bury earlier ones. The Mila / University of Montreal line of work (Goyal, Bengio, and colleagues) is useful here precisely because it does not treat an ordinary residual stream as a workspace. It implements an explicit shared global workspace as a limited-capacity bottleneck that coordinates specialized neural modules. That design choice is evidence that a generic residual bus is not already a workspace: if it were, the extra bottleneck would be redundant. Language models in the mainstream decoder-only family were not given that extra object. They were given a residual stream. Calling the residual stream a global workspace therefore smuggles a slot ontology into an additive vector channel.

A fair reply is that global workspace theory was always a functional sketch, not a claim about vector-valued slots, so an additive bus could still implement bottlenecked broadcast. That reply is coherent. It still does not license treating width as irrelevant. If broadcast is cashed out as linear availability to many readers, the rank of the bus remains a constraint. The residual stream can implement bottlenecked broadcast in some weaker sense, but the implementation constraint is the hidden dimension and the write contention along network depth, not a literary blackboard of independently editable propositions.

Skip-connection lore is the wrong frame for capacity claims. Residual connections were introduced so that gradients could traverse network depth without vanishing, and that remains a valid optimization story. It does not imply that the residual stream is a high-capacity working memory or a consciousness substrate. A skip connection explains why information can survive from early layers to the final layer. It does not explain how many independent items can be made simultaneously available to many readers without interference. Transformer residual streams are wide enough, in published models, to carry a great deal of latent structure. They are not wide enough, and not the right data type, to underwrite an unconstrained global workspace. The residual stream is a communication medium with a fixed rank. Treating vanishing-gradient folklore as a capacity theorem confuses trainability with workspace occupancy. Interpretability needs the former as background and the latter as a separate, falsifiable claim about residual stream geometry and readout.

Two further architectural facts keep the bus from becoming a temporally persistent, queryable workspace by default. First, the residual stream is local to a token position. Attention can move information across positions, but it does so by writing low-rank updates back into each token’s own residual stream, not by copying a shared slot file that all positions edit in place. Second, the only privileged linear readout of the residual stream at the end of network depth is the unembedding at the final layer. Intermediate residual states can be inspected with the logit lens, and the logit lens is genuine evidence that next-token information is linearly present before the final layer, but a linearly readable prediction is not a roster of workspace items. The residual stream is therefore a hard bus in two respects that matter for capacity: it has a fixed hidden dimension, and later writes can overwrite earlier ones. It is also indexed by token rather than by a durable workspace address. Any global workspace capacity claim that ignores those facts is a metaphor, not a mechanism.

What the capacity claim is actually asserting

A short, non-hype primer is needed because the phrase global workspace is often used in commentary on language models as if it named any shared hidden state. Global workspace theory, in the Baars package and in the later global neuronal workspace of Dehaene, Changeux, and colleagues, is not a synonym for “the system has internal activity.” It is a limited-capacity theory of access. Specialized processors run in parallel. They compete for entry into a shared workspace. A winning coalition is broadcast, and that broadcast makes the content globally available for report and for further routing. Dehaene and Changeux treat ignition and global availability as empirical signatures of that access event, not as proof of an unbounded store. Capacity in global workspace theory means limited simultaneous access. It does not mean unlimited latent storage in a high-dimensional substrate. That distinction is the entire reason residual stream width matters. A large language model can store a huge amount of statistical structure in weights and in unused residual directions while still having only a small number of contents that are simultaneously, globally, and reportably available.

The functional markers that come with the package are specific. Reportability says workspace contents can be used in deliberate report or control. Competition and broadcast say multiple candidates contend and winners are distributed widely. Capacity limits say only a small number of items occupy the workspace at once. Ignition says entry is threshold-like rather than a faint graded leak. Global workspace theory also distinguishes architectural questions from phenomenal ones: one can ask whether a system implements bottlenecked broadcast without answering whether the system is conscious. That distinction should be kept intact when the system is a stack of transformer residual streams. The residual stream is a candidate implementation substrate for some of those markers. It is not, by itself, evidence that all of them are present. Middle layers are often where commentators expect competition and broadcast to concentrate, because that is where representations are less embedding-like and less unembedding-like than at the edges of network depth. Expectation is not measurement. A global workspace claim has to say which marker is being asserted and which residual computation is supposed to implement it.

The standard analogy says layers or heads are specialists and the residual stream is the workspace. The analogy is attractive because the residual stream is in fact the shared medium, and because every submodule reads and writes that medium. It breaks as soon as capacity is taken seriously. Specialists in global workspace theory do not all write into a single additive vector whose rank is the hidden dimension. They compete for a bottleneck whose occupancy is small by design. Additive residual writes are not winner-take-all global availability. A feature can be written into the residual stream and remain locally useful to a later MLP without being broadcast in the GWT sense, and a later write can destroy that feature without anything like inattentional blindness having occurred. Uncritical posts that equate the residual stream with a broadcast channel skip this step. They treat the existence of a shared bus as if it already supplied competition, ignition, limited occupancy, and reportable global access. Those are extra claims. They may be true in some middle layers of some language models. They are not implied by the existence of transformer residual streams.

Even so, the analogy is not useless. If the claim is only that some contents are more widely readable after a contended write, residual-stream mechanics are a reasonable place to look. The error is promoting that coordinating function into an occupancy certificate. Capacity talk also gets inverted. Classical global workspace theory is interesting because the workspace is small. Mapping that theory onto an unbounded residual metaphor contradicts both the neuroscience source and the architecture. The residual stream of current large language models is wide compared with a 7-dimensional toy model and narrow compared with an unconstrained propositional blackboard. Published widths do not grow like the conceptual inventories people want to store in a workspace. If a commentator claims that a model implements a global workspace, the honest version of the claim is that some bottlenecked broadcast, with limited simultaneous occupancy, lives in the residual stream. The dishonest version is that the residual stream is a high-capacity theater of consciousness because it is shared. This article takes the first version as a hypothesis generator and rejects the second as a category error. Residual stream width constrains the first version. It does not need to be invoked against the second except to say that the second is not a capacity claim at all.

Bandwidth Versus Broadcast Along the Prediction Direction

A vector in R^{d} cannot be an unbounded blackboard. Linearly readable independent directions are at most rank-d without a superposition cost. That is an information-theoretic and linear-algebraic bound, not a training detail. If a global workspace capacity claim says that N independent items are simultaneously globally available, and if global availability is cashed out as linear decodability from the residual stream by many downstream readers, then N cannot exceed the hidden dimension unless those items share directions and interfere. Layer writes collide in a stronger sense as well: later residual updates overwrite or rotate earlier writes, so a “broadcast” at layer 8 is not automatically still present at layer 24. Attention does not repair this by instantaneous global copy. Attention is typically full-rank in d_model only as a sum of low-rank head writes. Each head communicates through a narrow bottleneck and deposits a low-rank update into the residual stream. That is limited-bandwidth communication. It is not a workspace-wide memcpy. Residual stream geometry is therefore the right object, and the wrong picture is a set of cleanly labeled slots that any specialist can read without paying interference or overwrite costs.

One useful coordinate on that residual stream geometry is the prediction direction: the unembedding column of the token currently assigned the highest probability. The logit lens methodology already showed that intermediate residual states are organized, to a surprising degree, around directions that become next-token logits. High variance directions in the residual stream are often poorly aligned with that prediction direction, which means the principal variance axes of a prompt corpus need not be the axes that control model output. A prediction-proximal band can be small even when the hidden dimension is large. That is a geometric reason, independent of any new benchmark, to stop treating raw width as workspace capacity. The counterpoint is that a small reportable band is compatible with a large packed substrate. Complexity can live off-axis. Global workspace theory, however, is about access rather than about that substrate, so the off-axis remainder does not enlarge workspace occupancy.

Token-wise and sequence-wise readings of “workspace” must be separated or the capacity claim becomes ambiguous. The residual stream at one position is a token-local state. A temporally persistent, addressable global workspace would require that the same contents remain queryable across generated tokens, or that a controller can fetch them by identity rather than by hoping they still sit in a nearby residual direction. Language models can approximate persistence with attention over the context and with features that are rewritten each step, but that is not free workspace memory. It is recomputation through the same width-limited residual stream. Middle layers are the usual place to look for such recopying, because middle layers sit far enough from the embedding to have mixed contextual features and far enough from the final layer to be more than a copy of the logit. The logit lens, applied along network depth, is then a probe of how early the prediction direction becomes linearly readable, not a census of workspace items. Sequential orthogonal projection off the prediction direction is one way to ask what else is linearly present after that rank-1 axis is removed. Sequential variance ordering of the remaining directions is one way to ask how many additional axes still carry prompt- or task-level variation. Those are analysis tools for residual stream geometry. They do not, by themselves, show that the residual stream is a global workspace. They do show that broadcast, if it exists, has to live in a vector whose usable rank is bounded by width. The most output-visible axis may be the prediction direction, and the contents are not automatically durable across time.

Superposition packs features rather than adding slots

Superposition is the default geometry when features exceed residual stream width. The Anthropic toy-model result is that n_features greater than d_model implies interference, not extra slots. Features are packed into overlapping directions so that a wider conceptual inventory can live in a narrower hidden dimension. The cost is that activating one feature moves the residual stream along directions that also encode others. Polysemantic residual directions cannot be globally available propositions in the global workspace theory sense, because a reader that decodes one item necessarily mixes others. That is the opposite of a clean workspace slot. Sparse autoencoders make the packing visible in production models: dictionaries extracted from the residual stream are far larger than residual width, which is evidence of packed latents rather than slot-like workspace items (Anthropic, 2024). SAE writeups sometimes treat dictionary size as richness of representation. The opposite implication is the one that matters for capacity claims. If the dictionary is much larger than the hidden dimension, the residual stream is not a slot workspace. It is a compressed medium under such packing pressure that extra “contents” are not independently addressable.

The tradeoff is real. Packing lets a model store more features than it has dimensions, which is why superposition is useful. Those extra features are not independently addressable, so they cannot be counted as extra workspace occupancy. Participation ratio and related effective-dimension statistics are then more honest than dictionary size: they ask how many directions are actually in play, not how many names an overcomplete basis can assign. Relatively few dimensions can dominate the linearly available, output-visible band even when the SAE list is long.

Instruction tuned models are the right place to keep this distinction sharp, because alignment training can change what is routed through middle layers without changing residual stream width. The hidden dimension of a Llama-class or GPT-class stack is an architectural constant. It does not grow when a base checkpoint is turned into an instruction-following checkpoint. If instruction tuned models look more workspace-like — cleaner instruction following, more stable intermediate reports — the a priori expectation should be better use of a fixed-width interface, not a larger interface. That claim should be checked on base models and on ten instruction tuned models from the same families, not on a single aligned checkpoint that happens to verbalize its steps. Ten instruction tuned models can differ in routing and in how often the prediction direction is occupied by task frames rather than surface bigrams. They cannot differ in the rank of the residual stream unless the architecture itself changed. Packing pressure therefore remains after alignment. Instruction tuned models may learn to compress more task structure into the same residual stream. They do not receive new orthogonal slots. Treating post-training as if it minted a larger global workspace is another form of the unbounded-blackboard error, only now applied to a training recipe rather than to parameter count.

Published widths already bound the popular stories

Published widths already bound popular capacity stories. A responsible walk is 512 to 768 to 1600 to 4096 to 8192 to 12288. Those numbers are hidden dimensions, not workspace occupancies. Even the largest of them is a rank bound on linearly independent, simultaneously available directions in one token’s residual stream. Cognitive-science explainers that mention transformers almost never confront these figures or the write contention across layers. They should. A claim that a model holds hundreds of independent, globally available propositions in one residual state is a claim that either superposition is costless or that d_model is being ignored. Neither option is compatible with the toy-model interference result or with the architecture. The narrowest residual stream in common use is already wide enough for rich features and already too narrow for an unconstrained blackboard.

One could object that a hidden dimension of 12288 is already a large working set if occupancy is counted in independent linear directions. That objection cuts both ways. It makes a modest, limited-occupancy GWT analogy more plausible, and it makes unconstrained blackboard stories less so. Scaling model width more slowly than claimed conceptual richness is a problem for unconstrained GWT analogies. Each family shows systematic expansion of total residual coordinates as models grow, and that expansion is real representational capacity. It is not automatic expansion of broadcast occupancy. Residual stream width as raw d_model and residual stream width as usable global access are different quantities. Only the second is a workspace capacity.

Attention deserves a separate line in the same table of constraints. A multi-head block can, in aggregate, span the hidden dimension, but it does so as a sum of head-sized updates. Broadcast, if the word is still used, is low-rank and contended rather than a clean global copy. Residual stream geometry in middle layers is then a record of many small writes, not a stage on which a full-width proposition is posted once and read by everyone. The prediction axis sharpens the same point for report. Model output is a function of a linear map applied to the residual stream at the final layer, and the logit lens shows that a similar map is already somewhat meaningful earlier. Contents that never align with that prediction axis can be causally important and still fail the GWT reportability test. High variance directions that point away from prediction can dominate a PCA plot while remaining weakly visible to the unembedding. Principal variance axes are therefore a poor proxy for workspace occupancy. Variance proximity to the prediction direction is a better descriptive axis than energy alone: it asks which residual directions are close to the readout, not which directions wiggle most across a corpus. A manifold complexity gradient along that axis is a coherent hypothesis about residual stream geometry. Near the prediction direction one can expect locally structured manifold geometry, including tightly folded local structure that keeps task or prompt groups locally clustered relative to flatter, prediction-distal regions. Farther away, lower values correspond to smoother, less linearly reportable variation. That manifold complexity gradient is a research program, not a completed census. It is already enough to reject the idea that every residual dimension is an equal workspace slot.

Interface capacity and substrate capacity should be named separately so that scaling stories do not blur them. Interface capacity is how many distinct, reportable items can be simultaneously represented in directions that downstream modules and the unembedding can actually use. Substrate capacity is how much background state can be packed into the rest of the residual stream, including directions that are only nonlinearly or indirectly visible. Global workspace theory, if it is going to be used at all, is about the first quantity. Scaling d_model mostly enlarges the second. Persistent rotational interventions are one way this split would show up empirically: a small rotation in a readout-aligned direction should move model output immediately, while the same angular budget spent in a prediction-distal band might degrade internal computation without a clean verbalizable change. Until such tests are done carefully, the honest statement is modest. Published model width sets a hard rank ceiling. Superposition and low-rank attention writes make the usable ceiling lower than that rank. The prediction axis and the logit lens suggest that the output-visible slice can be smaller still. None of those facts requires a new 2026 corpus of unpublished geometries. They follow from the architecture, from the residual-stream communication picture, and from published superposition and SAE results.

Thin-shell geometry is the last residual-stream fact that capacity claims should not ignore. If activations concentrate in a narrow norm band, then mean pairwise euclidean distance between residual states is driven by angle rather than by magnitude. Direction becomes the scarce resource. A workspace implemented as residual directions then has a directional budget, and adding coordinates to the hidden dimension mostly adds directions that need not lie near the prediction axis. Residual stream structure, in that thin-shell regime, is a story about angles and readout alignment. It is not a story about an ever larger set of independent magnitudes, each of which could hold one workspace item. Commentators who want “scale will provide the missing slots” owe a mechanism for moving those new directions into the reportable, globally readable band without paying superposition cost. Current large language models have not exhibited such a mechanism in the published circuit literature. They have exhibited a shared residual stream of fixed hidden dimension, plus dictionaries larger than that dimension. That is a bound, not a vibe.

Tests that would make the claim falsifiable

A metaphor becomes a mechanism only when it forbids something. Three tests are close to the original global workspace theory package and can be run on the residual stream of language models without new metaphysics. Competition: does a new write suppress unrelated residual features as if they lost a limited-capacity contest, or do writes simply accumulate until later layers happen to overwrite them? Broadcast: after a single write, is the same residual direction linearly decodable by many downstream modules — several heads and a later probe — rather than by one local reader? Persistence: does that direction remain queryable across subsequent tokens, or is the apparent workspace only a token-local residual state? Activation patching, causal scrubbing, and targeted SAE interventions are the right instruments because they ask about usable bandwidth, not about dictionary size. Persistent rotational interventions belong on the same list: if a small, sustained rotation of a candidate workspace direction collapses report or dual-task performance while a matched rotation of a control direction does not, the candidate has survived a causal test that slogans cannot pass. Middle layers should be the first place those interventions are applied, because that is where a bottlenecked broadcast, if it exists, is usually hypothesized to sit. The logit lens can serve as a cheap pre-screen for whether a direction is even report-adjacent, but the logit lens is not itself a workspace detector.

None of these tests is cheap, and a negative result would not prove that no weaker coordination story is true. It would only show that the strong GWT package is not sitting in the residual stream in the form claimed. Failure modes should look like the cognitive signatures that made global workspace theory empirical in the first place. If the residual stream implements a limited global workspace, dual-task collapse or an analogue of inattentional blindness should be producible by loading additional independent items into the same residual directions. If instead performance degrades smoothly as superposition interference grows, the better description is packed features on a width-limited bus. If ablating a small set of residual directions in middle layers destroys deliberate report while leaving fluent continuation intact, that is evidence for a privileged access band. If the same ablation is harmless unless it moves the prediction direction, the residual stream is organizing around next-token readout rather than around a workspace roster. Counts of simultaneously decodable features, and tests for whether extra residual writes suppress unrelated features, are admissible metrics. They are admissible only when they are measured, not when they are inferred from parameter count. Until those measurements exist for a given model, a global workspace capacity number is not a finding. It is a prior.

What to claim instead

Interpretability should replace “the model has a global workspace” with width-indexed statements about linearly available features per token. A responsible sentence names the hidden dimension and the layer band, plus the readout (probe, SAE decoder, or unembedding). It should also name the interference budget. It says, for example, that in middle layers of a model with hidden dimension 4096, a specified set of features is jointly linearly decodable from the residual stream before a later write destroys them. It does not say that large language models implement access consciousness because they have a residual stream. Global workspace theory remains a useful source of hypotheses about bottlenecked broadcast and limited occupancy. It is not a certificate of consciousness and not a certificate of unbounded working memory. The explicit workspace modules studied at Mila are the existence proof that if one wants a workspace, one can build a limited-capacity object on purpose. Mainstream language models built a residual stream instead. Instruction tuned models can be described in the same vocabulary: compare base and aligned checkpoints, including ten instruction tuned models when a family-level claim is at stake, and report whether alignment changes usable bandwidth or only the policy over a fixed-width residual stream. Network depth should be reported as the number of successive contended writes, not as a mystical deepening of awareness.

The same discipline applies to geometry talk. Residual stream geometry is a real research object. The logit lens and sequential orthogonal projection are legitimate tools for describing it, as is tracking the prediction direction. A hypothesized manifold complexity gradient can organize experiments about where task structure sits relative to readout. None of those tools licenses a capacity number that ignores d_model. SAE dictionary size should be cited as evidence of packing. Attention should be cited as limited-bandwidth communication. Token-local residual states should not be narrated as temporally persistent workspaces without a persistence test. If those constraints are respected, residual stream width becomes what it should have been all along: an architectural upper bound on how many independent items can be made globally available, and a reason to keep global workspace capacity claims small, falsifiable, and numerically honest.

Editorial Team

Editorial Team

The J-Space Editorial Team curates and synthesizes research in mechanistic interpretability and AI alignment. The team favors empirical rigor over slogan-level claims, and writes for researchers who already work with residual-stream evidence.