The Jacobian Lens, Explained

Most interpretability tools tell you what a model might be thinking. The Jacobian lens lets you change its mind and watch the answer shift — causally, not correlationally. It is the instrument behind Anthropic’s Global Workspace paper (Gurnee et al., 2026), and it deserves its own technical explainer.

A model can contain a decodable signal without using that signal to produce its answer. That distinction is the central problem for interpretability. A linear probe may recover a concept from an activation; a sparse autoencoder may isolate a direction correlated with it. Neither shows, on its own, that the direction carries causal weight. The stronger test edits the internal quantity and asks whether behavior changes in the predicted, selective way. The Jacobian lens was built around that test.

TL;DR

• The Jacobian lens reads, steers, patches, and ablates a transformer’s internal workspace using the model’s own averaged gradient map from each layer to the final layer.
• It extends the logit lens by replacing the identity matrix with the expected Jacobian of the final output, so its readouts can be tested by intervention rather than trusted on faith.
• Coordinate-swaps on J-lens vectors flipped verbal reports, multi-hop arithmetic, and planning strategies — selectively, without breaking adjacent capabilities.
• The lens is limited to single-token concepts, noisy through roughly the first third of the model, and needs white-box access; a behavioral analogue for external memory is sketched here only as a research direction.

A Jacobian is a first-order map describing how a small change in one internal quantity propagates to another. For a transformer, the Jacobian of the final-layer residual stream with respect to an intermediate hidden state measures the local linear influence of that intermediate computation on the final output.

The Jacobian lens is the logit lens with the identity replaced by the model’s own averaged gradient map — the expected Jacobian of the final output with respect to each layer’s residual stream. That single substitution turns a brittle, correlational readout into an instrument you can intervene with.

From the logit lens to the Jacobian lens

The J-lens sits at the intersection of two questions, and keeping them apart prevents several category errors:

1. Was the mapping trained from labeled or output-matching data, or derived from the model itself?
2. Does the method expose a correlation, or does it support a causal intervention?

The lens stands on a four-rung ladder, each rung adding rigor along one of those axes.

Logit lens (nostalgebraist, 2020). Multiply a layer-ℓ hidden state by the unembedding matrix W_U and softmax to a vocabulary distribution. The distributions converge roughly toward the final answer, but the mapping is ad hoc — the workspace paper frames it as “setting J_ℓ = I in our formulation,” the identity standing in for the true derivative. Unsupervised, but brittle.

Tuned lens (Belrose et al., 2023, arXiv:2303.08112). A per-block affine probe, trained on a frozen model to make intermediate predictions more reliable than the raw logit lens. It stays correlational — optimized to match the output, not to intervene. Gurnee et al. observe that on unverbalized computations the tuned lens “tends to skip ahead to the output rather than surface those intermediates” — that is the workspace paper’s characterization of the tuned lens, not Belrose et al.’s description of their own method.

Causal tracing and ROME (Meng, Bau, Andonian, Belinkov, NeurIPS 2022, arXiv:2202.05262). Causal mediation analysis localized factual recall to mid-layer MLPs at the last subject token, and Rank-One Model Editing then altered those facts. This was causal, but scoped to single relations — proof that targeted internal edits change behavior, not yet a general-purpose workspace instrument.

Hernandez et al. relation-Jacobians. The immediate predecessor used Jacobians to derive per-relation linear maps (“plays instrument” → Miles Davis → trumpet). The J-lens applies the same first-order approximation to the whole map from activations to outputs, not a single relation. A linear probe needs labeled directions (supervised); the J-lens derives its directions from the model’s own gradients (unsupervised).

Gurnee et al. call the result “the principled correction” to the logit lens: J_ℓ is “precisely the average linear map that relates layer-ℓ directions to their final-layer counterparts.” The causal axis — not the derived-versus-trained one — is what earns trust.

How the Jacobian lens is constructed

For each layer, the method estimates the expected first-order Jacobian of the final residual stream with respect to the residual stream at that layer:

J_ℓ = E_{t, t′≥t, prompt}[ ∂h_final,t′ / ∂h_ℓ,t ]

The expectation averages over a source position t, all subsequent positions t′≥t, and a corpus of prompts, obtained by backpropagating from the final residual stream to h_ℓ. The result is one d_model × d_model matrix per layer, mapping layer ℓ into the final layer’s coordinates.

The released procedure uses “one thousand prompts sampled from a pretraining-like distribution”; the companion repository specifies 1000 prompts of 128 tokens and notes the averaged map saturates at roughly 100. The result is a reusable layer-to-output map — a first-order approximation, not a full reconstruction of the network’s nonlinear computation.

The four operations: READ, WRITE, PATCH, ABLATE

A single per-layer matrix J_ℓ enables four operations, useful because the readout and the interventions share one representation.

READ (lens readout). Project a hidden state through the Jacobian, normalize, and unembed:

lens(h_ℓ) = softmax(W_U norm(J_ℓ h_ℓ))

This yields a ranked vocabulary distribution — what the lens “sees” at that layer — without the raw logit lens’s assumption that the intermediate activation already lives in the final layer’s coordinates. Readouts can be explored in the released interactive demo.

WRITE (steer). Add a scaled J-lens vector v_t — a direction in J-space (the set of activations expressible as sparse combinations of J-lens vectors; made precise below) associated with a concept — to the residual stream:

h ← h + α v_t

The parameter α sets the scale. Steering tests whether the direction can influence the model rather than merely predict its output. But steering alone is not sufficient evidence of a clean mechanism — large perturbations cause broad side effects. The relevant question is whether the intervention produces the predicted change selectively.

PATCH (coordinate-swap). Express the activation in the coordinates of two J-lens vectors v_s and v_t:

V = [v_s v_t], c = V†h (where V† is the pseudoinverse), h_patched = h + V(σ(c) − c)

Here σ is the swap of the two coordinates (optionally α-scaled), not a generic nonlinearity; the component of h orthogonal to the selected vectors is left untouched. This is more targeted than adding a direction — it asks whether replacing one coordinate with another causes the matching behavioral substitution: “soccer” for “rugby,” “spider” for “ant.”

ABLATE. No closed form exists. Identify the k = 10 most strongly activated J-lens vectors and zero out the residual stream’s projection onto each, across a chosen layer band, removing the contribution of those workspace directions.

The J-lens vectors themselves are called subframes — token-indexed subsets of the model’s feature directions. J-space is defined by a fit rather than an explicit factorization: given an activation h_ℓ, gradient pursuit solves for a sparse, nonnegative combination of at most k ≤ 25 J-lens vectors that best reconstructs it. Sparsity and nonnegativity describe the fit, not the subframe. J-space is the set of points expressible this way — geometrically, a union of k-dimensional cones. The two k values are unrelated: ablation uses top-10; the J-space fit uses sparsity ≤ 25.

What the Jacobian lens demonstrated

The lens is an instrument, not a findings paper. The substantive results — the workspace and its memory-like properties — live in the Global Workspace coverage; here, only the demonstrations that validate the instrument matter.

Coordinate-swaps flip verbal report (Soccer → Rugby; a Spanish-language report of García Márquez → a French-language report of Victor Hugo). They change multi-hop reasoning: swapping “spider (8 legs)” to “ant” makes the arithmetic answer “6” instead of “8.” A rhyming-couplet plan swap (Fight → Light) shifts earlier words, and an A/B bandit strategy flips. The interventions are selective — adjacent capabilities stay intact. The models were Sonnet 4.5 (primary), Haiku 4.5, Opus 4.5, and limited Opus 4.6.

The methodological lesson is narrower and more durable than any single result: an interpretability claim earns trust when someone edits the internal quantity and observes the predicted behavior change with selectivity. A decodable correlation shows only that information is present. A successful, controlled intervention shows that the channel is load-bearing.

Honest limits of the Jacobian lens

The lens is powerful but tightly bounded, and the authors are explicit about what it cannot do.

Single-token scope. The lens “only identifies vectors associated with concepts that correspond to single tokens in the model’s vocabulary.” Multi-token concepts such as “San Francisco” and “machine learning” have no single unembedding direction; an appendix offers only partial extensions. The limit is structural: the vocabulary indexes the lens’s directions.

Small integers — cause unresolved. Small integers fail to swap reliably, and the paper leaves the reason explicitly open: either “the model may compute over small integers outside the workspace,” or “its working representation of small integers may simply not align with the J-lens vectors.” The current evidence does not select between those hypotheses.

Imperfect and early-layer-blind. In roughly the first third of the model the readouts are noisy and largely uninterpretable; interpretable ones concentrate in mid-to-late layers. Whether that early silence is genuine absence or lens degeneracy is left open — “it remains possible that parts of the model’s true workspace, not captured by the J-lens, operate in earlier layers.” The authors’ own hedge is broader still: “The Jacobian lens is an imperfect tool, which we believe only approximately and incompletely captures the model’s underlying workspace structure.”

Narrow experimental scope, and circularity. The phenomenon is shown only within a single forward pass at intermediate token positions — no claim of persistence across separate calls. All experiments are on Anthropic-internal Claude models, with no SAE or attention-circuit corroboration. And the lens is built to find verbalizable content, then evaluated for further properties — a circularity worth naming.

Two widely repeated numbers deserve scoping. The technical paper reports hub connectivity only qualitatively — J-lens vectors “compose with downstream weights more broadly than other representational vectors”; the “factor of about a hundred” figure appears in Anthropic’s plain-language overview, not in the paper itself. And the dramatic ablation collapse (near-ceiling to near zero) belongs to the controlled multi-hop reasoning eval; on GSM8K math, chain-of-thought solving is substantially more robust to ablation than answering the same problems directly — the model externalizes reasoning onto the page.

External instrument, not a runtime capability. J-space is legible only via a lens researchers apply from outside the network; the model has no channel to read its own workspace and cannot introspect J-space. External inspectability should not be mistaken for a runtime introspection capability — it is architecturally absent, not merely undiscovered. This is the core reason self-report is not an audit.

What a lens for external memory would mean

This section is our framing — a research direction, not a Mnemoverse product, not a shipped feature.

The white-box Jacobian lens cannot be reproduced on a black-box model like a production assistant: there is no access to weights, activations, attention, or per-token logprobs. The no-weights analogue for an external memory system is behavioral causal mediation.

Run the reader with an injected memory atom present (full), removed (null), and minimally corrupted (swap), judging the answer each time. If the atom truly entered the working set, the full condition differs predictably from the null and swap arms; an atom that sits inert while the model answers from its parametric prior is not load-bearing. This is a natural-indirect-effect design applied to retrieval — a working-set verifier yielding per-atom labels (load-bearing, redundant, ignored-inert, harmful, contested), an evaluation instrument, not part of the memory runtime.

The caveats are honest ones. The null arm is ambiguous: an atom can look ignored either because it was unused or because the answer was overdetermined by the model’s prior. The only proposed mitigation — probe facts the base model cannot know — is not yet built, and no source quantifies the real-world reliability of a derived Memory-Influence Score. The analogy stops at a research question, not a method: can external-memory systems adopt the J-lens’s strongest norm — controlled, selective intervention — without model weights? That is the gap between storing a fact and proving it was used, the discipline AI agent memory will need.

Common questions

What is the Jacobian lens?

The Jacobian lens is a causal interpretability instrument that uses a transformer’s own averaged gradient map (the Jacobian of the final output with respect to each intermediate layer) to read, steer, patch, and ablate internal workspace vectors. Unlike probes or SAEs, it is tested by direct intervention on the model’s computations.

How does the Jacobian lens differ from the logit lens and the tuned lens?

The logit lens decodes hidden states with the unembedding matrix, which the workspace paper frames as setting the Jacobian to the identity: unsupervised but brittle. The tuned lens trains per-block affine probes to match the output distribution, improving reliability but remaining correlational. The Jacobian lens replaces the identity with the model’s own averaged Jacobian, so its directions can be tested causally: editing a J-lens vector changes the output in the predicted, selective way rather than merely correlating with it.

What are the four operations of the Jacobian lens?

READ: project a hidden state through the Jacobian, normalize, and unembed to get a vocabulary distribution. WRITE: add a scaled J-lens vector to the residual stream. PATCH: swap the coordinates of two J-lens vectors in the activation’s representation. ABLATE: zero out the projection of the residual stream onto the top-10 most active J-lens vectors across a chosen layer band.

What are the main limitations of the Jacobian lens?

It is limited to single-token concepts, fails to swap small integers reliably (cause unresolved), produces noisy readouts in roughly the first third of the model, and operates as an external instrument the model cannot introspect. The authors describe it as an imperfect tool that only approximately and incompletely captures the workspace structure.

Can the Jacobian lens be applied to black-box models?

No. The lens requires weight-level access to compute Jacobians and intervene on activations. For black-box models, a behavioral analogue using full/null/swap retrieval interventions is proposed here as a research direction, but it is not a validated method.

How does the Jacobian lens relate to causal tracing and ROME?

Causal tracing (Meng et al., NeurIPS 2022) localized factual recall to mid-layer MLPs, and Rank-One Model Editing (ROME) edited those facts. The Jacobian lens generalizes the causal intervention to any output direction, not just a single relation, by averaging the Jacobian from each layer to the final layer over many prompts.

Related

• Global Workspace Memory, Outside the Model — the findings the Jacobian lens enabled.
• AI Introspection: Voice, Not an Audit — why external instruments beat self-report.
• AI Agent Memory — the memory-for-agents landscape.

Last updated 2026-07-11.