A single counterexample outweighing a stack of proofs on a balance
Security · Episode 2

Part I — The one asymmetry underneath all of it

An exploit is an existential claim that carries its own witness. A security property is a universal claim over a model you cannot verify from inside the system. Uniform capability acceleration does not scale those two things equally.

Key takeaways
  • Offense needs one witness against the real system; defense needs a proof against a model of it, and the model is not verifiable from inside.
  • Measurement, training, and authorization all require a witness-bearing predicate — which is why the same capabilities resist all three for one structural reason.
  • Uniform capability acceleration is not neutral: it scales existential search faster than it scales the construction of adversary models.
Season 4 · Beyond Verifiable Reward · Episode 2 of 13

Part I

The one asymmetry underneath all of it


If you only read one episode of this season, read this one. Almost everything that follows is a consequence.

An exploit is an existential claim. There exists an input that produces this behaviour. It comes with its own certificate: run it. The witness is the proof, verification is execution, and nobody has to trust the person who found it.

A security property is universally quantified. For all inputs, this behaviour does not occur. There is no finite object you can hand someone that settles it. You cannot run a .

That is the asymmetry. It is old — older than any of the technology in this season — and it explains a surprising amount.


The version of this claim that is too strong

You will find people, including me in earlier drafts, stating it like this: security properties carry no witness and cannot be verified.

That is false, and it is false in a way that a formal-methods reviewer will produce in one line. Non-interference is a -property with a decision procedure on finite systems. Information-flow type systems establish -properties by construction. Cryptographic reduction proofs establish -properties under stated assumptions. seL4 exists. Constant-time-code verification exists. If universally quantified security properties genuinely admitted no verification, an entire productive discipline would be a hallucination.

So here is the bounded version, which is both true and sharper:

The Bounded Asymmetry

Security properties are universally quantified over an adversary model. Within a stated model they can be proved. The model itself is not verifiable from inside the system.

The asymmetry survives the correction. It just relocates, and it relocates to a more interesting place. Offense needs one witness against the real system. Defense needs a proof against a model of it. The residual risk does not live in the proof. It lives in the gap between the model and the system — which is, more or less, a definition of the job.

That relocation is the whole of security engineering. Every attack you have ever admired was, structurally, a demonstration that the model omitted something. Rowhammer said the memory model omitted physics. Spectre said the ISA model omitted microarchitecture. Every parser differential says the specification model omitted a second parser.


Why three different problems are one problem

Take three things people complain about separately.

01

Measurement

We cannot benchmark the capabilities that matter most — conjecture quality, definitional fecundity, explanatory power. Every proposed metric turns out to score something adjacent.

02

Training

We cannot optimize for them either. Reinforcement learning from verifiable reward works spectacularly where the reward is a cheap mechanical check, and stalls where it is not.

03

Authorization

We cannot write policy about them. A policy engine needs a predicate it can evaluate on a request, and "is this agent doing something conceptually dangerous" is not one.

These are usually treated as three fields with three literatures. They are one structural fact seen from three angles: each requires a computable, witness-bearing predicate evaluable within budget. The capabilities in question are -shaped over infinite futures. They carry no witness. So they resist measurement, training, and authorization — for exactly one reason, not three.

That unification is worth more than it looks. It means a breakthrough in any one of them is a breakthrough in the others, and it means proposals that solve one by assuming the others are solved are circular. A meta-verifier that adjudicates conceptual value is not a new control; it is the same missing predicate with a model wrapped around it.


Acceleration is not neutral

Here is where the asymmetry stops being philosophy.

Suppose capability improves uniformly — the same lift for everyone, attacker and defender alike. That sounds balanced. It is not.

The attacker's task is existential search under a cheap, mechanical oracle. Did the exploit fire? The program crashed or it did not. Constraint solving, symbolic execution, fuzzing an execution path, searching a gadget space — all of these have exactly the structure that automated search is good at, and all of them self-verify on success.

The defender's task splits in two. Proving a property within a model is automatable, and that side genuinely accelerates. But constructing the model — deciding what the adversary can read, write, influence, and observe; noticing that memory has physics; noticing that there are two parsers — is not existential search under a cheap oracle. It is the thing that has no oracle at all.

The Automation Split

So uniform acceleration scales -search fast, scales in-model verification fast, and leaves model construction where it was. The offense-defense balance shifts not because attackers get better tools but because the defender's bottleneck was never in the part that got faster.

You can predict from this which defensive work will and will not benefit from automation, and the prediction is checkable. Verification, fuzzing your own code, differential testing, invariant inference — these accelerate. Threat modelling, trust-boundary identification, and the judgement about what your assumptions omit do not, and if anything they get harder as systems get more complex faster than your model of them does.


Assurance deflation

There is a name for the resulting condition and it is worth adopting: assurance deflation. Deployed capability scales quickly. Formal assurance scales slowly. Under competitive pressure organizations deploy the capability anyway, and the objective security margin shrinks relative to the capability even though nothing visibly breaks.

Nothing visibly breaks is the point. Assurance deflation has no incident, no alert, and no line in the risk register. It shows up as an organization whose documented control set is unchanged while the thing being controlled has moved two orders of magnitude. The controls are still there. They are just controlling a smaller and smaller fraction of what the system now does.

Episode eight is about a specific, quantifiable instance of this: defence stacks whose documented layer count has grown while their effective layer count has not.


What to do with this

Write down your adversary model and date it

Not the threat model for a system — the adversary model itself: what can this party read, write, influence, and observe? Most organizations have never written it down, which means it cannot be reviewed, cannot be versioned, and cannot be found to be wrong.

Audit the model, not the controls

Control audits are -proofs within a model, and they will pass. The interesting question is which assumption in the model is doing the most work and what happens when it is false. Rank your assumptions by blast radius on failure. That list is usually short and usually surprising.

Stop treating automation lift as symmetric in planning

When you evaluate a security tool that promises capability improvement, ask which side of the split it lands on. If it accelerates in-model verification, budget for it — it works. If it claims to accelerate model construction, ask how, specifically, because that is the claim that is almost always wrong.

The rest of the season is instances. Grindability, escape-adjacency, vacuous certificates, correlated controls, illusory comprehension — each is a different place where the model omitted something and the omission was the whole story.

beyond-verifiable-rewardseason-4ai-security formal-methodsthreat-modelingoffense-defense-balance