Verification contracts: teaching your repo how to prove itself

The gap: proof that never gets written down

Metatron's job so far has been to capture how an agent should think about a repository — the intent, the invariants, the conventions — as curated markdown it consults before writing code. It says nothing about how to prove a change works when you run it. And that's a real gap, because the run-and-verify knowledge is exactly the kind that lives nowhere durable.

Watch what happens today. An agent lands a change, writes a unit test or two, and then — when a human or the next agent asks "how do I actually run this?" — reconstructs the operational steps inside its prompt window: bring the services up, apply the migrations, seed the fixtures, fire a request, check the status code and the response shape. None of that is committed. When the session closes it's gone, and the next agent rediscovers it from first principles. It's the same rediscovery problem Metatron already solves for architecture, only pointed at verification. Repositories have no durable, machine-readable, human-reviewed artifact that answers "how do we prove this behavior, and what does a failure imply?"

What a verification contract is

A verification contract is a git-tracked OKF markdown file that lives right beside your decisions, under context/verification/. It reuses the machinery decisions already run on: the same frontmatter and scope: model that binds a contract to a subsystem, the same reserved-filename discipline, the same mirror import/export path. A contract has a handful of explicit sections the parser keys off — the body stays human-readable and diffs cleanly:

• Assumptions — the pre-existing state to verify before anything runs (a database reachable, an env var set).
• Setup — the commands that stage the world: migrations, seed fixtures.
• Checks — each an action plus its expected assertions (a status code, a JSONPath value, a substring), tagged so you can select a slice like smoke or critical-path.
• Failure Means — the part that doesn't exist anywhere else.
• Teardown — put the world back, whether the checks passed or not.

Because a contract is scoped to one subsystem and indexed like a decision, an agent working on auth pulls the auth contract and nothing else. That's a deliberate echo of what our context inheritance study found about delivery: selection beats volume. A single monolithic testing.md read into a context window crowds out the task the same way a decision monolith did — it landed below the no-context baseline while costing the most tokens. Scoped, served on demand, never concatenated. That selective serving is the whole reason to build this inside Metatron rather than leave it a loose docs/testing.md convention.

The differentiator: "Failure Means"

Every test runner tells you a check went red. None of them tells you what red implies. That's the section we care about most. Failure Means is a curated mapping from a specific red check to the subsystem that is probably at fault — so a failure routes to the right fix instead of sending someone off to guess.

Concretely, for an auth contract it reads like this: a 401 on the happy path points at a seed hash mismatch or a missing user, or a JWT_SECRET that isn't set; a 500 on the wrong-password path means the auth middleware is leaking an unhandled exception; a connection refused means the gateway is down and setup never completed. No existing tool — not pytest, not Postman, not Playwright, not Bruno — encodes what a failure implies about which subsystem. It is a curated decision about failure modes, the same kind of distilled operational judgment Metatron already stores for architecture, and it's what lets a reading agent turn a red result into a directed fix.

Who authors it: the agent that just built the feature

Here's where the design leans directly on something we published. In the context inheritance study, the single variable that decided whether repository context helped was the capability of whoever wrote it relative to the agent that reads it. Context is worth most when it's authored by whoever holds the most capability and the most context. For a verification contract, that author has an obvious identity: the agent that just finished building the feature, at the exact moment it holds the most context it will ever have about how the thing works.

So metatron verification setup wires a one-time instruction into AGENTS.md: after finishing a testable feature, draft a verification contract describing how to prove it works and what a failure implies. The just-built agent is its strongest possible author. But — and this is the invariant we hold everywhere — a draft is never self-canonical. It lands through the same review gate as decisions: either straight onto a working branch where the reviewed pull request that merges it is the curation act, or staged as a candidate a human promotes with a reviewed git mv. A human placing or approving the file into verification/ is the curation act. Nothing crosses that boundary on its own.

Running it: executable-first, and selective

The contract is inert markdown until someone runs it. That's metatron verification run — a plain developer-and-CI command. The first milestone is deliberately boring in the best way: executable-first. It evaluates the declared assertions deterministically — status and exit codes, substrings, JSONPath, regex, with a shell-assertion escape hatch — with no model in the loop. It's cheap, offline, and it exits non-zero on failure, so it drops into CI and pre-commit unchanged.

It's selective by design, the same lesson as serving, one level down. With no scope it runs every canonical contract; narrow it with --scope or --tags and CI can run just the smoke and critical-path checks on every pull request while a nightly job runs the full set. Run beats a monolith. And when a check goes red, the report prints that check's Failure Means line right next to it — the differentiator carried all the way through to the output, so the person (or the agent reading the report) routes the failure instead of guessing.

There's a later, opt-in --judge mode on the roadmap for invariants no assertion can express — "the error copy stays user-neutral," "the migration is reversible" — where a model judges a check against the contract's intent. It's explicitly opt-in, clearly labeled non-deterministic, and never the default or a merge gate on its own. Phase one ships without it.

The honest fence: Metatron serves, the operator runs

Now the part we want to be exact about, because it's easy to get wrong. Metatron runs against private codebases with an on-prem, offline posture — your source never leaves the machine. Given that, execution has to sit on the right side of a trust boundary, and we draw it hard.

Metatron's serving path never executes a contract. There is no run_verification MCP tool, no runner behind the request path. On a private on-prem deployment, a primitive that let the serving layer pipe markdown-defined shell commands into a runner would turn any writable context/ file — or a prompt-injected candidate — into an arbitrary-code-execution vector reachable over the wire by any agent. So the serving surfaces stay strictly read-only: an agent can fetch a contract to read, never to run.

run is a different trust boundary entirely. It's a command a developer (or a CI job they configured) invokes in their own shell, foreground, with their own privileges — identical in trust to running the commands or pytest by hand, which they could always do. And it's readable before it's run: --dry-run resolves and prints the ordered plan and the resolved assertions without executing anything, so you can see exactly what a contract would do before you let it. Two rules keep the fence honest: nothing an agent reaches over MCP ever runs, and only a human or a CI job they set up ever triggers execution. An agent-authored or injected contract is inert until a person chooses to run it — the same human gate that guards canonicalization guards execution.

Where this is headed

Phase one is live in Metatron 0.13.0 — the metatron verification commands and the read-only MCP tools — with the --judge mode still ahead. The shape is consistent with everything Metatron already is. A repository should carry not just how to think about itself, but how to prove itself: a curated artifact, authored by whoever holds the most context at the moment of most context, served selectively to whatever agent shows up next, and executed only by a human or CI on their own trusted side of the wire. The durable value stays where Metatron always puts it — in the reviewed artifact and, above all, in the curated reading of what a failure means.