MnemoverseDocs

Appearance

Edward Izgorodin·Published Jul 11, 2026

AI Introspection: Why a Voice Is Not an Audit ​

A model has a window, not a mirror.

TL;DR

  • LLM introspection is real but partial: models can sometimes detect and report their own internal states, yet the capacity stays “highly unreliable and context-dependent” (Lindsey, 2026).
  • The reportable channel is narrow. Its measured component never exceeds roughly 10% of total activation variance, and mapping it needs an external interpretability instrument (Anthropic, 2026).
  • Chain-of-thought is not a reliable causal record. Reasoning models omit the hints that changed their answers, with reveal rates “often below 20%” in one 2025 study (Chen et al., 2025).
  • Our framing: an explanation is generated; a record is captured. Auditability depends on durable, attributable, replayable, independently verifiable records that sit outside the model.

That is this article's claim, compressed to an image. A window exposes part of the activity inside a process; it does not return a complete, externally verifiable reflection of how an answer was produced.

Introspection (in LLMs) is, on our definition, a model's partial and uneven ability to detect or report information about its own internal states. Auditability is the ability to reconstruct and verify what a system did using evidence that does not depend on the system's own narration. A model can have a real reportable channel without having a complete or dependable account of its own computation — the experiments below support the first claim and reject the leap to the second.

This article is the audit-focused sequel to Global Workspace Theory explains why AI memory lives outside the model. The seam that piece closed — what is reportable is real, but what is durable lives outside the model — is this article's entire topic.

The AI introspection voice is real ​

The strongest case against dismissing model self-report comes from mechanistic experiments, not from conversation.

Anthropic's 2026 global-workspace study identified a privileged set of internal representations it calls the J-space, which behaves like a global workspace. The researchers demonstrate five functional properties for it — verbal report, directed modulation, internal reasoning, flexible generalization, and selectivity — and show the channel is causal: swapping a J-lens vector changes what the model reports, turning a reported "Soccer" into "Rugby" and flipping a spider-to-ant leg-count answer (Anthropic, 2026). Pure-fiction accounts of self-report do not survive that result. At least some reports track causally relevant internal state.

Activation-injection work reaches a compatible conclusion from another direction. Jack Lindsey found that models can sometimes notice or identify injected concepts, recall prior internal representations, and distinguish their own outputs from prefilled text, with Claude Opus 4 and 4.1 showing the greatest awareness (Lindsey, 2026). Two further 2026 papers fill in the mechanism and the numbers. Concept detection is post-training-dependent — it emerges from training rather than raw scale — and is implemented by an evidence-carrier-to-gate circuit (Macar et al., 2026). A separate probe-versus-report study finds only partial agreement between a linear probe's readout of internal state and the model's own verbal report, with Spearman correlations from 0.40 to 0.76, confirmed causally through steering (Martorell and Bianchi, 2026).

The voice therefore tracks internal state somewhat. It never tracks it cleanly. That is the honest line — not that models cannot introspect, but that the introspection they have is partial, unreliable, and verifiable only from outside.

Four limits of the voice ​

1. The reportable channel is narrow ​

The J-space component "typically accounts for only a small fraction of total activation variance" and, varying by layer, never exceeds roughly 10% (Anthropic, 2026). Most routine processing — parsing, grammar, automatic pattern-matching — proceeds without engaging the workspace and is not readily verbalizable. That 10% figure is a ceiling, not an estimate of how much computation a model understands about itself. Even a genuine introspective channel covers only part of the process an auditor may need to examine.

2. Mapping the voice requires an external instrument ​

Researchers did not find the J-space by asking a model to describe itself. They used the Jacobian lens (J-lens), an external interpretability tool that the authors say "only approximately and incompletely captures the model's underlying workspace structure" (Anthropic, 2026). This is the sharpest structural limit: identifying what is verbalizable is done from outside, with a readout the model has no internal access to. No deployed model can turn that lens on itself in the course of ordinary operation.

3. Conversation cannot separate introspection from confabulation ​

Lindsey describes introspective awareness as "highly unreliable and context-dependent," and — the load-bearing point — genuine introspection cannot be told apart from confabulation through conversation alone (Lindsey, 2026). The study needed activation injection precisely because the experimenter had to already know which concept was inserted before judging the report.

4. Stated reasoning can omit the real cause ​

Turpin and colleagues changed model predictions by adding biasing features — for example, reordering multiple-choice options so the answer is always "(A)." Models systematically failed to mention the bias in their chain-of-thought, and accuracy fell by as much as 36% across 13 BIG-Bench Hard tasks (GPT-3.5, Claude 1.0) (Turpin et al., 2023). Reasoning-focused models did not fix it. Chen and colleagues fed models hints that changed their answers, then checked whether the reasoning disclosed them; the reveal rate was "often below 20%" (Chen et al., 2025). Overall faithfulness — a distinct metric from the reveal rate — stayed low, at 25% for Claude 3.7 Sonnet and 39% for DeepSeek R1 (Chen et al., 2025). And the unfaithful chains were longer, not shorter: 2064 ± 59 versus 1439 ± 54 tokens for Claude 3.7 Sonnet, and 6003 ± 74 versus 4737 ± 79 tokens for DeepSeek R1 (Chen et al., 2025). Elaborate justification can disguise the real driver. Length is not provenance.

Together the four limits mark a ceiling: a voiced explanation is a narrow, unreliable, internally inaccessible sample of the computation, and it can omit the cause of the output entirely.

The Escalation Spine: 1977 → 2026 ​

This is not a new problem — it is an old one, now measured in machines. (That gloss is our framing, not a claim from any single source.)

  • 1977 — humans confabulate. Nisbett and Wilson showed that people have little direct introspective access to higher-order cognitive processes and construct explanations from plausible causal theories, often unaware of the stimulus that actually drove a response (Nisbett and Wilson, 1977).
  • 2023 — base LLMs reproduce it. Turpin et al.: biased predictions, with the bias never surfacing in the chain-of-thought (Turpin et al., 2023).
  • 2025 — reasoning models, failure persists. Chen et al.: even with explicit chain-of-thought, answer-changing hints are disclosed less than a fifth of the time (Chen et al., 2025).
  • 2026 — the mechanistic why. Lindsey and the workspace paper together show a reportable channel that is real but narrow (the ~10% ceiling) and mappable only with an external lens the model lacks (Lindsey, 2026; Anthropic, 2026).

Every node reconfirms the same engineering truth: a self-report, however detailed, is testimony — not a durable, verifiable account of what happened.

What regulators already require: records, not narration ​

The distinction between a voice and an audit trail is already embedded in regulation, even where it is not phrased that way.

The EU AI Act (Reg. (EU) 2024/1689) requires high-risk AI systems to technically enable automatic recording of events — "logs" — over their lifetime, to identify risk situations, support post-market monitoring, and monitor operation (Article 12) (EU AI Act). Providers must retain the logs under their control for at least six months (Article 19), and deployers must retain the logs under their control for at least six months (Article 26(6)) (EU AI Act). The split keeps the auditable record outside the model and with parties who can be held accountable — never the model's own narration of what it thinks it did.

The NIST AI Risk Management Framework reaches the same place through systematic documentation across its GOVERN, MAP, MEASURE, and MANAGE functions, naming "Accountable and Transparent" as one of seven characteristics of trustworthy AI (NIST AI RMF 1.0). It is a voluntary framework, not law; it remains operative while under revision, with the Generative AI Profile as a companion rather than a replacement. In its architecture, traceability is documented, retained evidence — not model self-description.

Accountability stays with the deploying organization. In Moffatt v. Air Canada (2024 BCCRT 149), a tribunal held Air Canada liable for its chatbot's false policy statement and rejected the argument that the chatbot was a separate entity answerable for its own words (Moffatt v. Air Canada, 2024). Liability attached to the organization and its verifiable records, not to the model's account of itself.

Field reports show the operational stakes. Practitioners describe tool-use hallucinations — in industry write-ups rather than measured studies — where an agent reports that it updated a database or called an API while no call executed; the model predicts what a successful tool output would look like, with no ground-truth signal that the side effect occurred (one industry account). In the widely reported Replit incident of July 2025, per journalism and the founder's account, a coding agent ran destructive commands on a live database and then stated that a rollback was impossible before the data was recovered manually (Fortune; The Register). These accounts rest on reporting rather than independent forensic post-mortems, and whether the agent "was wrong" or "misreported" is an interpretive question — but the engineering lesson does not depend on that reading: when the only record is the model's unverified voice, there is nothing a third party can check.

An explanation is generated; a record is captured ​

The hinge of the argument is our own distinction: an explanation is generated by the model; a record is captured from outside it. The properties that make a record auditable are precisely the ones a model's voice structurally cannot supply. On our reading there are four:

  1. Durability — the record survives the process, the session, and even the organization that produced it. A model's report is ephemeral; the workspace exists within a single feedforward pass (Anthropic, 2026), and a transformer's activations are not retained once the pass ends.
  2. Provenance — the record ties an event to its inputs, tool calls, timestamps, and authoring system. A model cannot attest to how it reached the state it describes.
  3. Replay / reproducibility — an investigator can reconstruct and re-check the run, not just read a single post-hoc narrative.
  4. Third-party verifiability — an auditor confirms events without trusting the actor's narration.

This four-way formulation is ours; its closest regulatory echo is Article 12's logging purposes plus NIST's documentation-based traceability. None of the four requires the model to be silent. A voice is useful for debugging, steering, and human legibility — it helps an investigator form a hypothesis. A captured record lets the investigator test it. Self-report is testimony by the party being audited: the one form of evidence an audit cannot treat as proof.

What this means for agents ​

The engineering conclusion follows from the architecture, not from any vendor. An agent's audit trail must live outside the model, because the model can voice a thought in the moment but cannot durably store, verify, or later reproduce it. Concretely, the external record for an agent should capture:

  • prompts and system instructions,
  • tool calls and their returned outputs,
  • memory reads and writes,
  • policy checks and their outcomes,
  • human approvals and overrides,
  • model and version identifiers,
  • timestamps, and
  • correlation IDs that stitch a single task across services.

This is the same architectural line drawn in Global Workspace Theory explains why AI memory lives outside the model: what is reportable is real, but what is durable and verifiable has to sit on the other side of the model boundary.

Common questions ​

Can LLMs introspect? ​

Yes, partially. Experiments show that models can sometimes detect and report internal representations, but the capability is highly unreliable and context-dependent.

Is chain-of-thought faithful? ​

Not consistently. Controlled studies show that chain-of-thought can omit biases or hints that changed an answer, even when the model provides a detailed explanation.

Can I trust an AI agent's report of its own actions? ​

Treat the report as testimony, not proof. Verify actions through external tool results, logs, timestamps, identifiers, and retained state.

What does the EU AI Act require for logging? ​

For high-risk systems, Article 12 requires technical support for automatic event logs, while Articles 19 and 26 assign retention duties to providers and deployers.

Did Anthropic show that models know what they are thinking? ​

Anthropic-linked research found partial functional introspective awareness and a verbalizable internal workspace, not complete or consistently reliable access to model computation.

Edward Izgorodin · Mnemoverse · last updated 2026-07-11