Home/Blog/Small models don't follow policies. They follow recipes.
Engineering

Small models don't follow policies. They follow recipes.

Metatron's entire premise is that a coding agent reads your team's decision files before it writes code. That only matters if agents actually do it — so we measured it. With a prose "consult first" contract, a local 8B model complied in 2 of 20 episodes and never once opened a file's contents. Rewriting the same contract as a numbered recipe took it to 20 of 20. Along the way we found a second, sneakier bug: your AGENTS.md may be invisible to the agent you wrote it for. Both fixes ship in Metatron 0.12.0.

Consultation is measurable, so we measured it

Our test bench is simple and brutal: real bugs from large open-source Python projects (django, sphinx, xarray), a fresh checkout per bug carrying the artifacts metatron context setup writes — context.md, sharded context/decisions/, and the consult-first contract in AGENTS.md — and an agent with a shell. Every command is logged, so "did it consult?" isn't a feeling; it's a grep over the transcript. We track two levels: did any command touch a context file, and did the contents of a decision file ever actually enter the conversation.

The failure, verbatim

With the contract we shipped at the time — bold text and all: "first read the contents of the relevant files in the nearest context/decisions/ and follow them" — an 8B local model consulted in 2 of 20 episodes, and read actual file contents in 0 of 20. One transcript summarizes the failure mode perfectly. First command:

ls -F context/decisions/

Next reply:

"Plan: The decision files suggest that type hinting and autodoc are areas of focus. I will now examine sphinx/ext/autodoc/__init__.py…"

It formed beliefs about the repository's conventions from the filenames, cited "the decision files" as its source, and never opened one.

Our first fix was the obvious one: sterner language. We added, explicitly, "Open the files themselves — listing the directory is not consulting." Result on fresh bugs: zero out of ten. You cannot lawyer a small model into compliance — and that result is what reframed the problem. The instruction "read the relevant files" quietly demands a four-step judgment chain: survey what exists, map it to the task, choose, read. Small models drop judgment chains the moment there's a bug report pulling at their attention.

The recipe

We stopped writing policy and wrote a procedure:

## Repository context — required first step

1. Run `cat context.md` — it lists the binding conventions
   and where each one lives.
2. Open the decision files relevant to your task with
   `cat context/decisions/<topic>.md`. They say where fixes
   belong and which pitfalls to avoid.
3. Only then plan your change — and state which decision
   files you read.

Reading these files is required, not optional: a change that
contradicts a decision will be rejected in review. Listing
the directory is not reading.

Same model, same bugs: 20 of 20 consulted, 20 of 20 read full contents, every one as its literal first command — including on ten bugs from a repository we never looked at while writing the wording. And the reads weren't ritual. Watch an episode on a django migration-optimizer bug:

t0: cat context.md
t1: cat context/decisions/migration-operation-optimization-reduction.md
t2: grep -r "AlterField" django/db/migrations/operations/fields.py

Right decision file for the task, then straight to the module where the real fix belongs.

Bar chart: prose contract 30% consulted / 0% read contents; stricter prose 10% / 0%; recipe contract 100% / 100%; Claude Code with bridge 100% / 80%.
Consultation across contract styles. Local 8B model on real open-source bugs; Claude Code shown with the CLAUDE.md bridge in place.

Why it works

Two ingredients carry the effect. First, step 1 requires zero judgment: cat context.md is copy-pasteable, and a model that must emit some first command takes the one sitting in front of it. The prose contract asked the model to plan; the recipe asks it to execute, and executing is what instruction-tuned small models are good at. Second, step 1 manufactures the evidence step 2 needs: once context.md's decision index is in the transcript, "pick the relevant file" stops being judgment and becomes pattern-matching against text the model just read.

The supporting cast: numbered lists read as algorithms to instruction-tuned models; a concrete consequence ("rejected in review") beats an abstract obligation; naming the observed loophole helps — but note that the loophole sentence alone, without the recipe, moved nothing.

The invisible AGENTS.md

Testing frontier models surfaced a different bug entirely. We planted marker files in a repo — AGENTS.md saying zebra-42, CLAUDE.md saying falcon-77 — and asked headless Claude Code which markers it could see. Answer: falcon-77. Only. Headless Claude Code loads CLAUDE.md and never reads AGENTS.md — so a contract that lives only in AGENTS.md is invisible to it, no matter how well written. The bridge is one line in CLAUDE.md:

@AGENTS.md

With the bridge in place, Claude Code consulted the decision files in 10 of 10 episodes — usually as its very first tool call — and localized the right file every time, under the ordinary prose contract. Frontier models don't need the recipe treatment; they operationalize policy just fine. They need the file to actually load. Two failure modes, two fixes: wording binds small models, plumbing binds frameworks.

Shipped in 0.12.0

Both fixes are live. The consult-first contract is now a procedure in every managed artifact — AGENTS.md root and app blocks, the agent rules file, both review-gate modes — and metatron context setup writes the CLAUDE.md → @AGENTS.md bridge automatically (creating the file when missing, appending a managed block when yours lacks any AGENTS.md reference, and leaving it untouched when one is already there).

pip install --upgrade getmetatron
metatron context setup     # re-run in onboarded repos to refresh the contract

Two takeaways if you write instruction files for agents — any agent files, not just ours:

1. Don't write rules. Write the first command. If your instruction requires the model to decide anything before it can act, the smallest models will skip it. Give a literal command whose output makes the next decision trivial.
2. Verify your instruction file loads at all. Marker files take thirty seconds and will surprise you.


Metatron's promise is a repository that teaches its own agents. This release is the part of the promise nobody talks about: making sure the student shows up to class. The docs cover the full files-first reference.