Arithmetic Patching Paper: GN vs STR

How GN and STR are defined on the arithmetic prompts

The paper under the reader question is Zhang and Nanda, “Towards Best Practices of Activation Patching in Language Models: Metrics and Methods,” arXiv:2309.16042v2, dated 17 January 2024. It is not Stolfo, Belinkov, and Sachan (2023). Zhang and Nanda follow that paper’s basic-arithmetic setting and state that Stolfo et al. used the same STR corruption; the GN comparison is Zhang and Nanda’s.

Activation patching here is a causal intervention, not a correlation. The clean run caches activations. The corrupted run uses a damaged prompt. The patched run repeats the corrupted prompt but restores one cached clean activation. The patching effect is how much that restoration moves an evaluation metric back toward clean-run performance. In the arithmetic appendix, every experiment patches a single MLP layer’s activation at the last token of the prompt. That is not the sliding-window, last-subject-token MLP patch used in their factual-recall experiments.

The model is GPT-J, the 6B decoder-only model of Wang and Komatsuzaki (2021). Prompts are two-shot: two completed examples precede an incomplete third equation whose operator is +, −, or ×. Zhang and Nanda say Stolfo et al. found that this format improves accuracy; they do not report their own arithmetic accuracy. Because large integers split across tokens, operands are not sampled over all three-digit integers. For addition and subtraction they are drawn from {1, 2, …, 250}; for multiplication, from {1, 2, …, 23}. They first draw 200 prompts and discard those on which the model’s top-ranked output token is incorrect. The retained count is not reported, so every arithmetic curve below is conditional on GPT-J already being correct at rank 1, on an unstated subset of those 200 prompts.

Two reproduction limits sit in that definition. First, Appendix C names the noised positions but does not recompute or restate ν, so the 3× standard-deviation scale is the paper-wide GN rule, not a scale measured in the arithmetic appendix text. Second, the paper writes N(0, ν) and also N(0; ν) while defining ν as 3 times a standard deviation; it does not say whether the second Gaussian argument is a variance or a standard deviation. Equal sequence length is part of the general STR definition, but Appendix C only states the same-set integer draw, not a per-pair token-length check.

The metrics that make the later gap readable are defined paper-wide. Let cl, *, and pt denote the clean, corrupted, and patched runs, and let r be the clean answer.

• Probability patching effect: P_pt(r) − P_*(r).
• Logit difference: LD(r, r′) = Logit(r) − Logit(r′). The patching effect is LD_pt(r, r′) − LD_*(r, r′), normalized by LD_cl(r, r′) − LD_*(r, r′), so it typically lies in [0, 1], with 1 meaning fully restored performance and 0 the corrupted run. Appendix C says they compute logit difference but does not restate this normalization separately.
• They also compute equation (2), a metric devised by Stolfo et al. (2023), from the same last-token MLP patch. In prose, Zhang and Nanda identify its first term as a relative change whose denominator is P_*(r). The PDF text extraction does not preserve equation (2)’s typesetting cleanly, so this article does not treat a reconstructed closed form of the second term as verified. The printed equation is in Appendix C of the v2 PDF.

What they report for addition and subtraction

Focused on logit difference and probability, Zhang and Nanda write that they “observe gaps between GN and STR for addition and subtraction.” The quantitative summary in prose is one sentence: “STR is found to provide sharper concentration, up to a magnitude of 4×.” They point to Figure 6 (addition) and Figure 7 (subtraction). Each figure has three panels: logit difference, probability, and Stolfo et al.’s metric. Figure 8 is the multiplication control, where “GN and STR provides nearly matching results.”

The paper does not publish the two peak patching effects whose ratio is “up to” 4×, does not say whether that factor is the maximum over both operations and both metrics, and does not identify which MLP layer carries the peak. “Up to” means the largest concentration gap in that summary, not that every layer, or both operations, differs by 4×. Multiplication is the explicit bound on overgeneralizing: the same model, intervention, and metrics do not show a comparable GN–STR split for ×.

The direction is the point of the appendix. In factual association on GPT-2 XL, patching windows of MLP layers at the last subject token, they find the GN peak 2×–5× higher than STR across window sizes. Arithmetic on GPT-J is the counterexample they cite immediately after refusing to call GN peaks illusory: “GN does not always yield sharper peaks than STR. For certain basic arithmetic tasks in GPT-J, STR can show stronger concentration in patching MLP activations.” Those two ratios are not interchangeable. The 2×–5× result is windowed patching on a different model and task; the arithmetic runs are single-layer patches at the last token.

Equation (2) is a further, metric-specific observation, not a second estimate of the 4× gap. On that metric they say they qualitatively replicate results similar to Figure 2 of Stolfo et al. (2023) and find an “extremely pronounced peak with STR corruption.” Their stated mechanism is internal to the formula: the first term typically dominates the second because its denominator, P_*(r), “is usually tiny under STR corruption.” That small denominator “acts as a large multiplier that amplifies the absolute gap between patching different layers.” They note that “this effect is much smaller under GN since P_*(r) is usually not negligible.” No numeric value of P_*(r), and no multiplier other than the qualitative “large,” is reported for addition or subtraction. This is author interpretation of why the STR curves on panels (c) look extreme, not a measured causal effect beyond the patching intervention itself.

How the authors characterize that sensitivity

The arithmetic-specific characterization is not a claim that one corruption recovers the true circuit. After stating the addition and subtraction gaps, the reverse of the factual-recall peak, and the near-match on multiplication, they write: “This highlights that activation patching can be sensitive to corruption methods in a rather unpredictable way.” “Unpredictable” here means the sign and size of the GN–STR discrepancy are task-dependent in their own experiments, not that they fit a noise model or a confidence interval. The abstract’s broader claim, that varying metrics and corruption methods “could lead to disparate interpretability results”, is the paper-level frame; Appendix C is the arithmetic instance of it.

They keep three claims separate, and the text supports keeping them separate:

• Observed intervention result. On filtered GPT-J addition and subtraction prompts, single-layer last-token MLP patching produces different layer-wise concentration under GN and STR, with STR sharper by up to 4× on logit difference and probability. That is a causal-patching comparison under the conditions above, not a correlation and not an accuracy benchmark.
• Author reading of unpredictability. Because factual recall on GPT-2 XL went the other way, and multiplication did not show the same split, they do not treat “GN concentrates more” or “STR concentrates more” as a stable property of activation patching. They also explicitly do not claim that GN results in general are illusory or inflated.
• Author reading of equation (2). The extreme STR peak on Stolfo et al.’s metric is attributed to division by a usually tiny P_*(r), which amplifies absolute differences between layers. That explanation is not offered as the cause of the up-to-4× gap on logit difference and probability, and it is not evidence that STR is the correct circuit-localization method.

The discussion section’s preference for STR is a different argument, drawn mainly from indirect-object identification rather than from these arithmetic runs. They are concerned that GN puts the model off distribution by injecting noise never seen in training, and in the IOI circuit they give tentative evidence that Name Mover attention and a related value-patch fail to stay on the clean pattern under GN. They therefore recommend STR when analogous in-distribution tokens exist, and GN only when token alignment or lack of analogous tokens makes STR unsuitable. They do not repeat the IOI attention diagnostics on the arithmetic prompts, so that off-distribution mechanism is not shown for addition or subtraction in this paper. Applying the Section 6 recommendation to arithmetic is an extrapolation from their general argument, not a result reported in Appendix C.

Limits that the v2 text itself leaves open: experiments are on decoder-only models up to 6B; there is no reported external replication of the arithmetic GN–STR comparison; the post-filter prompt count, peak layer, and peak patching effects are unstated; equation (2)’s second term is not restated in prose; and the Gaussian parameterization of ν is not disambiguated. Figure 6 and Figure 7 are the addition and subtraction evidence for the concentration gap; Figure 8 is why that gap should not be read as a property of all three operators.

1. How GN and STR are defined on the arithmetic prompts
2. What they report for addition and subtraction
3. How the authors characterize that sensitivity