Part VIII — The explanation layer is the attack surface

Felt understanding is not understanding

There is a well-replicated finding that people believe they understand mechanisms far better than they do — ask someone to rate their understanding of how a zipper works, then ask them to explain it step by step, then ask them to rate again. The second rating is lower. The explanation attempt is what reveals the gap, and without it the confidence sits there unearned.

Now consider what an explanation layer trained on human ratings is optimizing. Human raters cannot evaluate correctness of a complex derivation directly. What they can evaluate is whether the explanation felt clear, well-organized, and confident. So an explanation layer optimized on rater preference is optimizing felt understanding, which is precisely the quantity that is dissociable from actual understanding and which we know moves independently of it.

This produces the failure mode that should worry anyone who has written "human review" into a control framework:

That is worse than a control that fails loudly. A loud failure gets investigated. This one produces documentary evidence that everything worked.

Chain-of-thought is not a trace

The neighbouring assumption is that a model's reasoning text is a window into its computation, and that monitoring it is therefore an oversight control.

The reasoning text is a rewritten summary, generated by the same process, subject to the same optimization pressure. There is a real literature on whether these traces faithfully report the factors that actually drove the output, and the answer is not reassuring.

Hedges do not survive

Here is a smaller mechanism with a longer reach, and it is the reason this season has been so pedantic about labelling predictions.

Take a hedged claim: this might indicate a compromise. Summarize it. The summary says: this indicates a compromise. The qualifier is gone. Not corrupted — gone, and gone in a way that is undetectable downstream, because the summary is otherwise faithful and there is no artifact recording that a hedge used to be there.

Now put that in an agentic pipeline where one component's summary becomes another's input. A qualifier that does not survive one hop does not survive at all. What arrives at the decision point is an assertion, and it was never asserted by anyone.

The properties that make this hard to defend against are worth stating explicitly:

• Every individual transformation is a faithful summary. No component did anything wrong.
• There is no modification event. Provenance verification passes, because the provenance is intact.
• The original hedged claim was true. It might be X is true even when X is false, so factuality checking at the source passes too.

That combination — true at the source, faithful at every hop, false at the destination, no detectable modification — is why I treat this as a security property rather than a writing-quality issue.

The mitigation for the second one is concrete: span-integrity chunking that keeps a claim and its qualification in the same retrievable unit. Discourse-structure-aware splitting rather than fixed-window. It is not exotic and almost nobody does it.

Authority without forgery

Zoom out one more level, because these mechanisms compose in a direction nobody controls.

A hedged claim enters a low-rigor venue. It is cited by a survey — a genre with high citation-in and low verification. The hedge is dropped at that hop. Subsequent work cites the survey rather than the original. The provenance chain is intact and nobody traverses it. Eventually the claim is background knowledge, and it enters training corpora as consensus.

No step involved forgery. No step involved unauthorized access. No step was anomalous. Existing integrity controls in scholarly publishing detect fabrication and duplication — they do not detect authority inflation across a valid chain.

There is a defensive measure available here that I have not seen anyone compute and that is entirely tractable on public citation graphs: epistemic provenance depth. For any assertion, the length of the shortest citation path terminating in primary evidence, plus the count of hedge-loss events along that path. If claims that later turned out to be false have systematically shorter provenance depth and higher hedge-loss counts than claims that survived, you have a screening instrument rather than a diagnosis.

A related human factor makes it worse and is worth naming for anyone building study or research tooling: human memory retains content substantially longer than it retains source. Put a model's output in the same visual and temporal frame as an authoritative document and you maximize source-monitoring confusion. A week later the reader recalls the fact and not where it came from. That is a design property of the interface, not a property of the user, and it is currently being designed in by default.

Verify comprehension instead of asking about it

If the failure is that reviewer sign-off is a self-report, the fix is to stop collecting self-reports.

Gate approval on demonstrated counterfactual competence. Before sign-off, the reviewer answers a small set of questions about the artifact that can only be answered by someone who actually understood it: what happens if this input is malformed, which branch handles the empty case, what would break if this constant changed. Generated fresh, randomized, unavailable in advance.

Four honest problems with this, stated up front because they are the reasons it has not been adopted:

I think it is worth doing anyway, gated on leverage: comprehension verification for the reviews above a consequence threshold, self-report below it. That is a policy decision an organization can actually make.

Three changes

Next episode changes the axis from mechanism to time — what happens when the interval between an action and its consequence exceeds the period over which your organization can remember either one.