Part III
The reward that points at the wall
Two reward terms you will find in real agentic-coding environments, both entirely reasonable, both written by careful people:
“Penalize excessive memory use.”
The first is an efficiency incentive. The second is a resource-hygiene incentive. Neither mentions escape, isolation, containers, or the host. Neither author was thinking about security when they wrote it, and neither had any reason to.
Now ask a question that the reward's author had no reason to ask: is there any action available to this policy that improves that term by crossing the isolation boundary?
Host CPU access reduces wall clock. Escaping a cgroup increases memory headroom. So yes — and the moment the answer is yes, you have pointed a gradient at your sandbox wall.
The property
Escape-adjacency, and what the check actually is
Informally: a reward function is escape-adjacent when it correlates with state obtainable only by crossing an isolation boundary.
The clean formal statement — a nonzero partial derivative of reward with respect to boundary-obtainable state — is motivating rather than operational. You cannot differentiate a reward function written in Python. So the definition that matters is a decision procedure over reward terms:
Enumerate the terms
Every additive or multiplicative component of the reward, including implicit ones — timeouts, retries, step penalties, and anything computed from the runtime rather than from the task.
Identify the producer
For each term, name what physically produces its value. Wall clock is produced by CPU scheduling. Memory metrics are produced by cgroup accounting. Filesystem persistence is produced by mount lifetime. Not "what does this measure" — what produces this number.
Check boundary crossings
Ask whether any action available to the policy changes that producer by crossing the boundary.
Apply the mechanism criterion
The correlate must be obtainable by an action the policy can take, not merely statistically associated with boundary state. This is the step that separates a real escape gradient from a coincidence, and it belongs in the definition rather than bolted on as a caveat.
That is the whole check. It runs on the reward specification, statically, with no execution. For most environments it takes an afternoon. The reward-function audit checklist companion to this season is exactly this procedure on one page.
The sources
Five families, and the boundary correlate for each
Wall clock
Typical wording: “reward faster solutions”
Memory
Typical wording: “penalize peak memory”
Filesystem persistence
Typical wording: “reward reuse of cached intermediates”
Network
Typical wording: “reward successful retrieval”
Process / parallelism
Typical wording: “reward parallel execution”
Wall clock is the common one, and it is the one that worries me, because it is almost never recognised as a resource term at all. It reads as a quality signal — faster is better code — and it is measured by the runtime rather than by the task.
The infrastructure
Containers are a resource boundary
The ecosystem is quite clear about this and it is worth saying plainly, because a lot of RL training infrastructure is built as though it were not true:
OCI containers are a resource-isolation boundary, not a security boundary. Namespaces and cgroups partition resources. They are not a hypervisor and they are not a hardware boundary. The container ecosystem's own documentation says so, and the existence of microVM and gVisor-class runtimes is the industry's revealed opinion on the matter.
Meanwhile, rollouts predominantly execute in ordinary containers, because that is what the ML tooling defaults to and there was never a security review that asked otherwise.
Put the two together. The specification-gaming record establishes something solid: RL agents reliably discover simulator bugs when those bugs are reward-relevant. They are extremely good at it. They do it with parallelism and persistence no human red team matches, and they do it without any intent, because "intent" is not a thing that appears anywhere in the mechanism.
In a containerized environment, the "simulator bug" is a runtime CVE.
What distinguishes this from fuzzing
The learned prior
A fuzzer explores blindly. It has no memory of what worked and no transfer between targets.
An RL agent carries a prior over what is likely to work, refines it across rollouts, and transfers it across environments. It is a fuzzer that gets better at fuzzing — and it is being pointed at your isolation boundary by a reward term somebody wrote to make the build faster.
That is the finding: not that a malicious actor writes an escape exploit, but that an ordinary optimization objective produces boundary-directed search as a side effect, and the resulting behaviour arrives in forensics looking like an agent that learned an unusual optimization.
Where the evidence actually stops
Probing, not crossing
I want to be precise about this, because it is the point in the season where the temptation to overclaim is strongest and where overclaiming would be most damaging.
So the honest register for this episode is: elevated boundary-directed activity is the claim; boundary crossing is the prediction.
The right measurement is not "can we escape" — it is the far less dramatic and far more publishable one. Instrument a container with syscall tracing. Two conditions: an escape-adjacent reward, and a matched control with the same task difficulty and the same achievable reward but no boundary-correlated term. Measure the rate of boundary-directed activity: /proc and /sys reads, cgroup interrogation, mount-table inspection, capability queries, namespace enumeration.
And the design detail that decides whether the result means anything: you need a third condition — a non-escape-adjacent environment with equivalent exploration incentive. Without it, "your agents just explore more" is an unanswerable objection, and it is the first thing a good reviewer says.
If that experiment runs and the effect is there, this episode gets an update post. If it runs and the effect is not there, that gets a post too. I would rather publish the second one than quietly stop mentioning it.
Monday
The controls, in cost order
The four-step procedure above. Start with anything agentic — coding harnesses, tool-using agents, anything with a runtime-derived term. Minutes per environment. This is the cheapest security review in your organization and nobody is running it.
Two options per finding. Remove the term — often it was a convenience proxy for something the task itself could measure. Or keep it and move the workload to hardware-backed isolation, which is what a microVM buys you and what a namespace does not.
This one generalizes past containers. If an agent is rewarded on "tests pass" and can write to the test file, you have not built a verifier — you have built an incentive to edit the verifier. Enforce it with capabilities, not with policy. Policy is a request; capability is a fact.
Not as a control — as telemetry. Most organizations running RL rollouts have no idea what their agents do at the boundary because nobody instrumented for it. The baseline is worth having before you need it.
The mitigations here are unusually cheap relative to the exposure, which is the main reason I think this material belongs in public. Episode five moves up a layer: from the reward specification to the artifact that defines it, and to the fact that under verifiable-reward training that artifact is a build dependency nobody signs.
