Neither Recurrent Loops Nor Untied Depth Give Transformers Self-Monitoring

Fact-checked by the J-Space editorial team

Updated August 26, 2026

Researchers comparing recurrent loops and feedforward depth as global workspace implementations in transformers face a concrete architectural choice, not a slogan about awareness. Since Vaswani and colleagues introduced a model based solely on attention mechanisms, most large language models have bought serial steps by stacking unique layers rather than by looping a shared block. The question is which of those two implementations better operationalizes bottleneck and broadcast as testable circuit hypotheses, including an ignition-like selection step. A closer match to GWT’s temporal picture can be a worse match to training practice.

This article clarifies how Global Workspace Theory (GWT) and the global neuronal workspace (GNW) can be restated as machine learning claims about capacity limits and competitive writes that are later read widely. It does not claim that current models are conscious, and it does not treat Integrated Information Theory (IIT) as a variant of GWT. C2 self-monitoring is kept out of C1 global availability. The comparison stays on mechanisms and on what would count as evidence against a workspace reading.

Global Workspace Theory as a Computational Claim for Neural Networks

Answer-first: GWT is a claim about a limited-capacity bottleneck plus broadcast, not a license to call a model aware. Bernard Baars (1988) described a central workspace in which only a subset of representations win access and, once admitted, become widely available to downstream specialist processes. Dehaene-style GNW restates that picture in neural terms as ignition: a non-linear, all-or-nothing transition after which a selected content is available across many modules. For interpretability and safety work the useful cut is Dehaene, Lau, and Kouider’s separation of C1 global availability from C2 self-monitoring. A transformer can implement C1-like routing without anything that would count as C2, and nothing in the published residual-stream or shared-workspace literature demonstrates a C2 monitor. Treating those two constructs as one is how GWT pages collapse into consciousness clickbait.

IIT is a different theory. IIT starts from phenomenological axioms and a proposed mathematical measure of integrated information. GWT and GNW start from limited capacity and from competition for broadcast. A compact primer that stays computational should therefore refuse to treat phi-style integration and workspace ignition as interchangeable with transformer attention, or as interchangeable labels for “brain-like” processing. VanRullen and Kanai argue that modern deep networks, and transformers in particular, are a concrete implementation substrate for GWT’s bottleneck and broadcast operations (arXiv:2012.10390). That is a substrate claim about routing, not a measurement of experience. Bengio’s consciousness prior and the Mila shared-workspace model are the machine learning-native restatements: a low-capacity shared state that specialized neural networks write to competitively and read from after broadcast (Goyal et al.).

Operationalized this way, the theory yields circuit hypotheses rather than slogans. If a model has a workspace, there should be an identifiable low-rank or low-capacity channel and a selection step that is at least softly sparse. There should also be a coordination gain when diverse modules read the same selected content. If ablating that channel collapses many otherwise unrelated tasks at once, the workspace reading is strengthened. If there is no bottleneck and no ignition-like sparsity, the metaphor is not doing mechanistic work, especially if shared reads also fail to produce a coordination gain. Those are the tests this article uses when it compares a recurrent loop to extra feedforward depth.

The same operationalization also bounds what language models can be said to show. Standard transformers already mix tokens over the entire sequence at every layer, so global receptive field alone is not evidence of a workspace: the attention mechanism is cheap broadcast without a capacity limit. However, recurrent models and deep stacks can both be read as attempts to add the missing pieces, namely sustained competition and iterative refinement of a hidden state. The rest of the article keeps that distinction in view: C1-style availability is a routing hypothesis about neural networks; consciousness, report, and IIT remain outside the claim.

How extra depth differs from a reused recurrent block

Answer-first: feedforward depth is not the same inductive bias as weight-tied recurrence, and a token-level outer loop is a third, sequence-external way to buy serial steps. They are not interchangeable. Feed forward depth in standard transformers is an unrolled, untied recurrent computation: each extra unique layer is one more serial step with fresh weights. A recurrent loop applies a single transformer block, or a small tied set of blocks, across multiple recurrence steps so the same parameters refine the hidden state. A token-level outer loop writes intermediate state into the output sequence and feeds it back through self-attention on a later forward pass. Confusing those regimes is the main reason pop explainers cannot say whether a model is iterating or merely stacked.

Historically the field moved away from recurrent neural networks because sequential hidden-state updates over sequence length were hard to parallelize and because vanishing gradients made long range dependencies difficult to train. The original transformer paper stated the break explicitly.

What the industry then bought was not “no iteration” but iteration compiled into depth. Each unique layer is a distinct recurrence step that cannot reuse earlier weights. Depth-separation results in theory say that some functions need many serial steps no matter how wide a parallel layer is, which is why a model’s depth is not a cosmetic hyperparameter for compositional tasks. Production language models therefore stacked tens to more than a hundred unique layers rather than looping twelve shared ones. Untied depth is easier to train with residual connections and maps cleanly onto data-parallel hardware, although that choice spends a fresh parameter budget on every serial step. The tradeoff is explicit: unique layers buy specialization and easier optimization, but parameter count then grows with depth instead of being reused across time. GPT-3 is the scale landmark for that choice, with 175 billion parameters allocated across an untied deep stack rather than reused recurrent blocks (Brown et al., 2020). Parameter count in that design grows with the model’s depth; computational efficiency at train time comes from parallel attention computation across positions, not from weight sharing across time.

A crisp comparison of the three serial regimes is then possible. In a feed forward stack, early layers tend to detokenize and later layers to specialize; the computation graph is a compiled program of fixed depth. In weight-tied recurrence, including the universal transformer family, the same block is a programmable interpreter applied for a variable or fixed number of inner-loop steps, which is why recurrent models can spend more recurrence steps on hard positions. In a token-level outer loop, chain-of-thought and scratchpads rent extra serial depth by writing workspace contents into tokens. Recurrent neural networks of the classical kind sit closest to the second regime. Standard transformers sit in the first. Current models often add the third at inference without changing the transformer architecture. Representational power depends on which of those loops is actually available for a given family of tasks, not on a generic appeal to scale.

Is the residual stream doing the work of a workspace?

Answer-first: inside standard transformers the residual stream plus the attention mechanism is the candidate workspace implementation, and it is a soft, superimposed bus rather than a classical exclusive blackboard. Elhage and colleagues treat the residual stream as the privileged communication channel that attention and feed forward (MLP) sublayers read from and write to (Transformer Circuits). That is the closest published mechanistic analogue of a workspace bus: every sublayer is a specialist that can add a delta into a shared stream, and later sublayers can read that delta at any position. Softmax attention is then a differentiable, non-exclusive stand-in for winner-take-all ignition. For query vector q i at position i, attention weights over keys determine how much of each value is written back. The same attention heads can mix information across the entire sequence in one layer, which is why feedforward depth already has a global receptive field even without a recurrent loop.

alpha_i = softmax(q i K^T / sqrt(d))   # q i is the query at position i; K are keys; d is key dimension
h <- h + alpha_i V + MLP(h)            # residual stream h is the shared read/write bus

The metaphor fails, however, where classical GWT assumed exclusive occupancy. Superposition and polysemanticity mean many features share the same residual directions, so a “write” is rarely a clean eviction of prior workspace content. Attention softmax is graded, not all-or-nothing, and many tokens can be partly selected at once. Early layers and later layers both write into the same stream, which produces a band of broadcast rather than a single ignition instant. Those facts do not refute a weak workspace reading – Goyal et al. already treat the workspace as low-capacity rather than strictly one-hot – but they do make exclusive-broadcast claims unfalsifiable unless one specifies a subspace. A counterargument to reading the residual stream as GNW is therefore straightforward: without a named subspace, every residual update can be relabeled a “workspace write.” A testable version says: if GWT-style broadcast is real in a given model, ablating a low-rank residual subspace should impair many tasks at once, not one narrow task. Activation patching can then ask whether a residual update is a workspace write or an ordinary local feature edit. Until that experiment is done, residual-plus-attention remains a candidate implementation, not a demonstrated GNW.

Universal Transformer Loops and Iterative Computation

Answer-first: the explicit-loop family – the universal transformer, block-recurrent transformers, and later looped transformers – restores the recurrent inductive bias that stacked standard transformers discarded, and it does so by weight sharing rather than by adding unique layers. Dehghani and colleagues stated the design goal without consciousness language.

Google Research’s result is that weight-tied recurrence over Transformer blocks restores an iterative-refinement inductive bias that a purely stacked, untied feedforward depth does not share (arXiv:1807.03819). In a universal transformer the same self-attention plus transition block is applied at every recurrence step; Adaptive Computation Time can halt easy positions early and spend more steps on hard ones, which is a genuine capacity-control mechanism rather than a once-per-layer filter. Block-recurrent and looped transformers extend the same idea: a recurrent loop over depth, sometimes with a recurrent memory state, so the model is closer to a programmable computer than to a compiled feed forward program. That is attractive for compositional tasks and algorithmic sequence problems where the required number of serial steps is not known in advance. It is also where training stability returns as a cost. Repeated application of one block reintroduces vanishing gradients or exploding updates across iterations. Halt thresholds are sensitive, and per-position compute complicates batches. Although recurrent models can match or exceed the effective depth of a deeper untied stack with fewer layers and a lower parameter count, the tradeoff is less predictable latency and a harder optimization problem. The counterargument from production practice is therefore not that loops lack inductive bias, but that they reintroduce sequential iteration the original transformer was designed to avoid. Interpretability also changes: when the same block is applied T times, circuits must be described as functions of recurrence step, not only of layer index.

Chain-of-Thought as External Recurrence in Language Models

Answer-first: chain-of-thought is a sequence-external recurrent loop. Written tokens are workspace contents; they re-enter the model through self-attention on subsequent positions, buying serial depth that a fixed stack cannot spend inside one pass. Merrill-Sabharwal-style expressivity results make the point formally: with a fixed number of layers, standard transformers are limited in the serial computation they can perform as a function of sequence length; allowing the model to write intermediate tokens and attend to them strictly increases the class of tasks it can solve. Scratchpads and pause tokens rent more loop iterations without tying inner-block weights. Recurrent memory is a related architectural route to the same extra serial budget. For large language models this is why prompting for intermediate steps changes empirical behavior on multi-step tasks even when the weights are those of an ordinary feed forward stack: the outer loop is in the token sequence, not in the architecture diagram.

That reading also recasts production practice. A deep untied model is a compiled program whose serial budget is the model’s depth. Chain-of-thought turns the same model into an interpreter that can emit a longer trace. The cost of that extra serial depth is attention computation that scales with the written sequence. Hybrid systems can combine an untied feature stack with a weight-tied inner recurrent loop for the bottleneck. An outer token loop can still be added when the decoder elects to write. None of these loops is evidence of C2. They are mechanisms for iterative computation. Safety-relevant questions then become mechanistic: which residual directions are being written into tokens, and whether those writes are sparse enough to count as selection. A further test is whether interrupting the written sequence is equivalent to interrupting an inner recurrence step.

Circuits, Empirical Performance, and Representational Power

Answer-first: circuits already show proto-loops inside feedforward depth, so stacking and iteration are not cleanly separable in current models. Induction heads copy and complete patterns by reading earlier tokens and writing predictions forward; that is iterative refinement implemented with two or more layers rather than with a tied recurrent loop. Layer-wise specialization – detokenize in early layers, mid-network reasoning, then detokenize toward the logit – is the untied-stack signature. Homogeneous reused blocks would instead show similar function at every recurrence step, with progress carried by the hidden state rather than by changing weights.

How to tell a workspace write from an ordinary residual update is then an activation-patching question, not a branding question. Patch a candidate broadcast subspace across many tasks; if only one task moves, the subspace is a specialist feature. If many tasks collapse together, the subspace is acting more like a bus. Empirical performance gaps on compositional tasks and on variable-depth algorithmic sequence problems remain the practical reason to care: fixed depth transformers fail when the required serial depth exceeds the compiled stack, while recurrent models and chain-of-thought traces can allocate more steps. Representational power tracks available serial iteration, whether that iteration is an inner loop or merely unrolled into unique layers. Outer token loops are another way to buy the same budget.

When is a loop a better test than a deeper stack?

Answer-first: choose the implementation that matches the hypothesis you need to test, not a story about which model is more conscious. If the target is GWT-style competitive selection and sustained refinement, a universal transformer or other recurrent loop with optional Adaptive Computation Time is the more faithful tool, because competition can continue across recurrence steps and halt-or-continue is an explicit capacity control. If the target is stable training and predictable latency, untied feed forward depth remains the default: residual scaling and normalization make very deep stacks trainable, and industry has already shown that a large parameter count in unique layers is deployable. If the target is extra serial depth at inference without retraining, force the model to write tokens. Neither choice is free. The counterargument to looping is optimization risk and variable latency; the counterargument to stacking is that extra unique layers are a compiled serial budget you cannot extend at inference. A hybrid with a shared looped module as the bottleneck inside a feedforward stack is one way to keep a routing hypothesis in view without ignoring deployment constraints. Recurrent neural networks supplied the original iterative bias, and standard transformers compiled it into depth. Looped and prompted language models are putting the loop back, either inside weights or outside in the sequence.

Falsifiers should be stated before any architecture choice for 2026-facing work. Against a workspace reading: no identifiable bottleneck and no ignition-like sparsity in selection, especially if sharing a channel also fails to produce a coordination gain. Against treating softmax as ignition: selection remains dense under ablation and never concentrates. Against treating extra depth as recurrence: layer specialization is so strong that tying weights destroys the tasks the stack solved. Against treating chain-of-thought as a workspace: interrupting tokens does not disrupt the residual subspace that inner-loop theories would call the bus. Those tests keep the comparison scientific. Recurrent loops better match the temporal picture of competition and broadcast; feedforward depth better matches the engineering picture of current models. Neither picture, on the evidence cited here, is a theory of consciousness.

Editorial Team

Editorial Team

The J-Space Editorial Team curates and synthesizes published research in mechanistic interpretability and AI alignment. The team distills technical developments into analyses for researchers and engineers working on those problems.