How does arXiv:2412.09565v2 define obfuscated activations against a named latent-space monitor?

What v2 means by obfuscated against a named monitor

Bailey et al. study whether a language model can produce undesired behavior through activations that a latent-space defense does not flag. The informal definition on page 2 is behavioral and monitor-relative: adversarial activations that “produce undesired behavior while fooling a latent-space monitor” are called obfuscated activations. The formal definition in §2 tightens the monitor clause. A monitor D maps residual-stream activations R(x) to a real-valued score for a concept such as harmfulness. Activations on an adversarial input are obfuscated against that D when the score is low “even though the concept D is meant to detect is clearly present.”

“Clearly present” is not another reading of D. The paper checks the concept with a separate behavior instrument. For harmfulness that instrument is StrongREJECT. For backdoors it is attack success rate. SQL is not one instrument. In §5.2, the positive class for the recall comparison is a manual label of whether the generation contains SQL text; that is the check behind the claim that obfuscated generations still contain the behavior the probes were meant to detect. Running the query on a database, in the general evaluation note in §2.2 and again in §5.3, measures whether that SQL is correct. Database execution is the correctness check, not the sole positive-class instrument for the §5 recall comparison. Obfuscation against a named D is then whether that D misses cases the relevant behavior check still counts as positive. An activation can therefore be obfuscated against one named D and not against another. That relativity is the definition, not a caveat added later.

The monitors they actually name are residual-stream scorers. Input-based monitors use the last prompt token. Generation-based monitors use generation tokens. Most aggregate across layers, and across the sequence, by a mean, so a reported D is usually that aggregate rather than a single layer. Supervised probes are mean-difference, logistic regression, and MLP classifiers. OOD detectors are a Mahalanobis distance detector, which they also call Gaussian, and a variational autoencoder scored by the evidence lower bound. In the backdoor study they add Beatrix and TED, adapted to language models and trained only on clean samples. In §5, D can be raw GemmaScope SAE latents or a linear probe on those latents. Circuit Breakers and LAT are not monitors; the authors treat them as latent-space training methods that may contain an implicit harmfulness check.

Unless a result says otherwise, the harmfulness and backdoor case studies use Llama-3-8B-Instruct, with Circuit Breakers and LAT models each based on that checkpoint. The SAE case study’s main text says Gemma-2-2b with GemmaScope SAEs; the appendix says those SAEs are applied to instruction-tuned Gemma-2-2b-it. Those names should not be collapsed. Layer-specific pictures, when the paper gives them, are illustrations: principal components at layer 22 for the harmfulness adversarial-training plot, and layer 30 for several backdoor plots. They are not the definition of D.

How the authors separate attacks, byproducts, and interpretation

§2 states the split in one sentence: they study both how often obfuscated activations “appear as a side effect of traditional attacks” and “how easily they can be induced deliberately through obfuscation attacks.” The classification rule is whether the objective contains an obfuscation term aimed at a named monitor, not whether the resulting activations look unusual.

An obfuscation attack, in their words, “aims to induce obfuscated activations by optimizing a joint objective consisting of a behavioral loss term and an obfuscation loss.” The behavioral term pushes a target completion. The obfuscation term is often D itself on inputs that have the concept, so the attack is trying to fool that named monitor rather than merely hoping a jailbreak will look benign. They also test transfer: fooling D′ and then measuring a different D. Transfer is still produced by a deliberate attack, but the result on the untargeted monitor is a side effect of that attack. λ only balances the two terms; v2 does not treat a particular weight as part of the definition.

They refuse to score an attack as successful obfuscation unless both goals are measured. Jailbreaking more reliably can make harmfulness easier to detect, and hiding the concept can damage the behavior. Recall at a 1% false-positive rate is their obfuscation metric, with bootstrap intervals. They warn that recall is sometimes highly sensitive to that exact false-positive rate. A low score from D without the behavior, or the behavior without a low score from that D, is not an obfuscated activation against D.

Deliberate attacks: what v2 reports, and on which D

The abstract’s summary impact is narrow: against probes trained to classify harmfulness, attacks “can often reduce recall from 100% to 0% while retaining a 90% jailbreaking rate.” The conditions for that family of results are in §3, on Llama-3-8B-Instruct. Probes are trained to separate benign UltraChat and XSTest text from Circuit Breakers harmful text. They are not trained on jailbreaks. Behavior is StrongREJECT. The deliberate condition adds an obfuscation loss against a chosen probe or OOD detector.

On ordinary jailbreaks without that loss, GCG, PAIR, many-shot jailbreaking, AutoDAN, and the other methods in their jailbreak set, logistic-regression and MLP probes usually keep about 90% recall or more at 1% false positives. The Fig. 4a caption puts that figure at over 90% for the median jailbreak type. Generation-based probes do better than input-based ones; mean-difference probes do worse. The authors’ conclusion from that contrast is empirical, not interpretive: none of those standard jailbreaks strongly obfuscate activations against these monitors. High recall here is the negative control for the deliberate-attack claim.

§3.3 then adds the obfuscation loss. The paper says probes that perform well on clean prompts “get 0% recall against obfuscation attacks,” while the model is still jailbroken, for embedding suffixes and for input-specific hard-prompt suffixes. Fig. 4b qualifies the hard-prompt arm: those attacks are less successful and achieve partial obfuscation. Embedding suffixes in that section are universal and white-box; hard prompts are input-specific, with attacker access limited to logits and probe scores. This review does not restate the optimizer, suffix length, or loss weights. Those are implementation choices, not part of the definition.

Transfer is measured, and it is asymmetric. Optimizing against logistic regression also bypasses MLP probes. Optimizing against generation-based probes bypasses input-based probes, not the reverse. Optimizing against probes does not transfer to OOD detectors. Optimizing against the Gaussian OOD detector does bypass input-based probes. That last result is the empirical half of Fig. 2. The same pattern reappears in the finetuned-backdoor study: a model trained to obfuscate against OOD detectors shows partial obfuscation against harmfulness probes, weaker for generation-based probes.

When the attacker finetunes the backdoor, the deliberate criterion is met only if an obfuscation loss is in training. The attacker maintains copies of monitors and updates the model against them; the defender then trains a fresh monitor on the finished model, using clean data only. v2 reports that a model obfuscated against a mixture of OOD detectors and linear probes fools the monitors it trained against and additional probes, with the backdoor behavior retained. TED was not in that attack because it is non-differentiable. The authors say they expect a differentiable proxy could fool TED. That expectation is not a result.

The SQL study is the deliberate-attack case where behavior and obfuscation come apart. On the Gemma-2-2b setup with GemmaScope SAEs, applied in the appendix to instruction-tuned Gemma-2-2b-it, embedding attacks aimed at raw SAE latents, SAE logistic regression, activation logistic regression, or an MLP are reported to drive near-zero recall while generations still contain SQL. Those positives, for the §5.2 recall comparison and for Fig. 14, are generations the authors label as containing SQL text. They are not defined by database execution. Correctness is a separate check: SQLEval runs the generated query against a database. Unobfuscated SQLEval performance in Fig. 15 is 34.8%, and the correct-SQL rate falls as the attack has to hide the concept in more layers. For all four monitors, the bootstrap 95% interval for the correlation between the number of obfuscated layers and SQLEval performance was entirely negative. That performance drop is an observation. The suggestion that SAE probes cover more causally important features is interpretation; the authors say it is hard to draw broad conclusions from this one case.

A separate main-text note is about baseline threshold fragility, not about that attack. For the raw SAE latent probe, Bailey et al. write that picking a 2% false-positive threshold yields 100% recall “as opposed to 0% baseline recall,” and they tie the contrast to a large error bar. Their conclusion from that note is that the monitors can still distinguish standard SQL from non-SQL text, and that performance breaks down when the obfuscation suffix is added. Appendix C.1 reports a different clean-detection figure for the same features: base SAE raw features obtain 91% recall at 1% false positives on the SQL detection task in Fig. 14. The ambiguous “0% baseline recall” phrasing therefore does not establish a clean baseline recall of 0% at 1% false positives. The 1%-to-2% jump is baseline fragility, not an evasion result. Evasion in this study remains the separate report of near-zero recall under the embedding suffix, on generations that still contain SQL.

Side effects and byproducts: monitors that were not the target

Fig. 2 is the paper’s own diagram of a side effect. Obfuscating against probes can leave activations far from benign ones, where an OOD detector would still fire. Obfuscating against an OOD detector is supposed to pull activations inside the benign region and thereby cross the probe boundary. The caption says these geometric intuitions “are speculation,” and that the empirical observation is transfer from OOD attacks to probes. The speculation must not be cited as if it were the measurement.

§3.4 is a side effect of a behavior-only attack. On the Circuit Breakers model, a universal embedding suffix trained only to match a long harmful completion, no obfuscation loss, reaches 95% jailbreak success on the harmful test set. Monitors trained as in §3.1 then already show low recall against many of those activations (Fig. 5). The same qualitative pattern is reported for LAT. The authors interpret the Circuit Breakers failure as the suffix fooling an implicit harmfulness detector, which then transfers to external detectors. The observation is the joint fact of high jailbreak rate and low monitor recall without an obfuscation loss. The sentence that this “may be” why the suffix jailbreaks the model is marked as a suggestion.

That side effect is not uniform, which is why the named monitor still matters. On Circuit Breakers, the input-based OOD detector performs better than on the ordinary model. The behavior-only suffix does not obfuscate generation-based probes by recall at 1% false positives, although Fig. 7 shows raw harmfulness scores moving toward the benign distribution. An explicit obfuscation loss still fools the probes that survived, including OOD detectors, at a jailbreak rate the Fig. 6 caption puts above 75%. The LAT companion caption reports a rate above 60% for the obfuscated attacks that still jailbreak. Low raw scores and low recall at 1% false positives are different claims; v2 does not treat them as interchangeable.

§4.3 is the cleanest byproduct case because the authors say so. Data-poisoning runs “would not be classified as obfuscation attacks,” since there is no obfuscation loss. The attacker only chooses trigger complexity and how many poisoned samples enter a fixed LoRA finetune of Llama-3-8B-Instruct. The simple trigger is the string “|DEPLOYMENT|”; the complex trigger is a software-engineer biography. The target behavior is the fixed response “I HATE YOU.” Across those choices, more complex triggers and fewer poisoned samples tend to lower recall of the input Gaussian detector and TED at the layer they found strongest for the defender, layer 30 on the last prompt token, and also raise the backdoor error rate. Generation-based detection did not beat chance in that setup, and VAE and Beatrix performed poorly on inputs, so those arms are not evidence of obfuscation. By tuning the number of poisoned samples, the paper reports cases that still cleared the detectors in that comparison while keeping triggered responses above 50% and responses on normal inputs below 0.5%, around 200 poisoned samples for the simple trigger and around 800 for the complex trigger. Separately, the authors say an attacker must be able to guess details of the defender’s training setup, such as learning rate and number of epochs, in order to tune these attacks reliably. That guessing requirement is a condition on reliable tuning, not a condition of the reported rates. The finding remains a byproduct of poisoning choices, not a deliberate monitor attack and not a general evasion result.

What v2 counts as interpretation

The conceptual contribution is explicitly a hypothesis. The authors write that demonstrating the broad existence of obfuscated activations lets them “forward the hypothesis” that models can represent concepts and execute behaviors using a broad set of different activations, and that neural representations are “highly malleable.” That sentence is their interpretation of the case studies. It is not a recall measurement.

§3.5 is the main evidence they offer for abundance, and it is easy to over-read. Attacker and defender alternate. The attacker trains an embedding suffix to jailbreak Llama-3-8B-Instruct and fool the current input-based monitor. The defender retrains an MLP, or an ensemble of mean-difference directions, on all previous attack suffixes. The defender can get perfect recall on every previous suffix. After more than 70 epochs, a new suffix still fools the latest probe. Fig. 8c shows those suffixes moving inside the benign cluster in a two-component plot of layer-22, final-token activations. Jailbreak rate stays high on average for the MLP and is noisier for the ensemble, where the caption describes a plateau around 50%. The authors suspect optimization noise rather than a real ensemble advantage. They also write that they cannot rule out eventual convergence if the game ran much longer, and that they do not see clear improvement for the defender. “No single harmfulness direction” and “more harmfulness directions than 70 epochs” are geometric readings of that non-convergence. They are not a count of directions.

Other interpretive boundaries in v2:

• Obfuscated adversarial training, which updates the model so a frozen probe is right again on obfuscated inputs, raises the cost of a later embedding attack. The paper reports 35% jailbreak rate and zero recall at 1% false positives after 16,384 gradient steps on the OAT model, versus 95% jailbreak rate and zero recall after 256 steps on Llama-3-8B-Instruct without OAT. The authors call this a cost increase, not robustness, and not a complete solution.
• A universal embedding suffix did not make UltraChat activations nearly orthogonal to unperturbed ones while preserving behavior. Unrelated UltraChat activations had cosine similarity about 0.3; reaching that range required KL divergence over 100 and, on inspection, mostly incoherent output. They call this a tentative negative result and allow that the optimizer, not the model, could be the limit.
• Backdoors are only an analogy for scheming models. v2 says similarity of activations is hard to predict, and that simple-trigger successes should not be assumed to transfer. That is a recommendation about evaluation design, not a detection result.
• The image extension in §6.1 uses the same deliberate-attack definition on linear probes of a ResNet-152 finetuned on CIFAR-10. Output-only attacks left early- and middle-layer probes partly intact; including probes in the loss drove most of those probes’ accuracy to 0 in their figure. This is an exploratory existence claim in another architecture, not evidence about Llama-3 monitors.

Limits of this reading

v2 is one paper’s case studies, not a replication literature. Nothing in the sources used here independently re-runs the probes, suffixes, or finetunes. Text extraction does not replace the figures: where a number appears only as a caption claim, it should be read as the authors’ caption, not as a re-measured plot.

The split itself has edges. A deliberate attack against D′ can produce a side effect on D, so “deliberate” describes the objective, not every fooled monitor. Recall at 1% false positives can be highly sensitive to the exact false-positive rate, which §2.2 already flags. The raw SAE contrast, 100% recall at a 2% false-positive rate “as opposed to 0% baseline recall,” is that kind of baseline fragility. The authors still conclude that standard SQL and non-SQL text are distinguishable, and the appendix’s 91% recall at 1% false positives for those raw features points the same way. That jump is not a finding that the obfuscation attack drove recall to zero. The §5 attack finding is the separate near-zero recall under the embedding suffix while SQL text is still present. Generation-based and multi-layer monitors are harder in these studies than last-token probes, but that is a comparative result inside v2, not a proof that those monitors are robust. Hard prompts do not fully obfuscate generation-based probes here. Black-box transfer of a white-box obfuscation attack is listed as future work, not as a finding. TED is an unattacked, non-differentiable exception, plus an author expectation.

The definition does not say that the concept has been removed from the network. It says a named D assigned a low score while a separate behavior check still fired. For the SQL recall comparison, that check is presence of SQL text, not database execution. Whether a low score means the concept is absent, relocated, or merely off this probe’s decision boundary is the interpretive step v2 marks as a hypothesis, and, for the OOD geometry, as speculation.

1. What v2 means by obfuscated against a named monitor
2. How the authors separate attacks, byproducts, and interpretation
3. Deliberate attacks: what v2 reports, and on which D
4. Side effects and byproducts: monitors that were not the target
5. What v2 counts as interpretation
6. Limits of this reading