Oracle First: Routing AI-Generated Diffs With a Glossary and Four Leaves

Consider this scene. It is a composite, not a personal war story.

An agent ran overnight on a leftover prompt. Morning git status showed fourteen files. Two of them implemented the requested endpoint. The rest were a new logger, a renamed helper, a rewritten Dockerfile, and a README that now contradicted the tests. Generation cost was close to zero. The next four hours were not.

That gap is the actual product problem. When a patch is cheap to produce, the expensive work is routing: keep, quarantine, or rewrite. Skip the routing and the cheap code becomes expensive debt with extra files attached.

This article is a glossary, a routing tree, and a worked example at each leaf. The artifact is a small classifier plus a quarantine command sequence. Treat the code as a proposal unless you run it on your own repo.

The problem the scoreboard hides

Free-model loops optimize for “a diff appeared.” Reviewers optimize for “this diff is safe to merge.” Those are different objective functions. A green unit test on a helper you did not ask for is not evidence that the architecture still holds.

Cheap generation also changes failure shape. The common failure is no longer “the model wrote nothing.” It is silent expansion: extra modules, extra dependencies, extra comments that drift from the contract. Routing has to detect that shape before anyone debates style.

Glossary

Use these terms as they are defined here. Nearby words in vendor blogs do not override them.

1. Cheap diff. A change whose generation cost is negligible next to the human time needed to decide its fate. Cost here means review and rollback, not GPU invoices.
2. Oracle. An automated check that can reject a patch without reading every line. A contract test, a typecheck, a golden fixture, or a linter with a frozen config can be an oracle. A vibe is not.
3. Surface area. The set of paths, exported names, and config files the diff touches. Count files, but also count contracts: HTTP shapes, CLI flags, schema versions.
4. Silent expansion. Files or refactors the prompt did not request. Expansion is a routing signal, not a taste argument.
5. Quarantine run. Execute the generated tree off the developer workstation, with no production secrets and no write access to the source remote.
6. Allowlist apply. Bring only named paths from the generated tree onto a local branch. Everything else stays discarded or parked.
7. Architecture-touching change. A patch that alters auth, persistence, process boundaries, or public contracts. These patches fail closed: rewrite, do not quarantine-and-hope.
8. Review budget. The hours a human can spend before the cheap patch costs more than writing it by hand. When the budget is exceeded, the tree’s answer is rewrite, not “one more prompt.”

The routing tree

Walk the questions in order. Do not skip to a leaf because the diff “looks small.”

Step 1 — Is there an oracle that already fails, or an oracle you can add in under fifteen minutes?

• No oracle, and you cannot add one quickly: go to Leaf D (rewrite the work, or rewrite the prompt into a smaller contract). Ungrounded generation is not a review task.
• Oracle exists or can be added: continue.

Step 2 — Is surface area bounded?

Bound means: requested paths only, or requested paths plus test files. A hard cap helps. A working default is “three production files and their tests.”

• Silent expansion beyond the cap: go to Leaf A (discard, tighten the prompt, regenerate).
• Bounded: continue.

Step 3 — Does the patch need network, secrets, or write access outside a temp directory?

• Yes, and the need is real (migrations, signed webhooks, vendor APIs): go to Leaf D unless you already have a dedicated staging path that is not your laptop.
• Yes, but the need is accidental (the model added telemetry, a download, or .env reads): go to Leaf A.
• No: continue.

Step 4 — Can an isolated process execute the oracle?

• Isolated execution is available (container, spare VM, or a throwaway server): go to Leaf B (quarantine run).
• Isolated execution is not available, and the patch is a pure function with an allowlist of one or two files: go to Leaf C (local allowlist apply).
• Isolated execution is not available, and the patch is larger than that: go to Leaf D.

The tree is deliberately biased toward discard and rewrite. Cheap generation makes “try it locally” the risky default, not the brave one.

Artifact: a proposed classifier

The script below does not prove safety. It only encodes the tree’s cheap heuristics so a human does not re-litigate them every morning. Label: proposal, unexecuted on your tree until you run it.

#!/usr/bin/env python3
"""classify_patch.py — proposal heuristic, not a security scanner."""
from __future__ import annotations

import subprocess
import sys
from pathlib import Path

ALLOWED_PREFIXES = ("src/", "lib/", "tests/", "test/")
ARCH_HINTS = ("auth", "middleware", "migration", "dockerfile", "compose", ".github/")
SECRET_HINTS = ("os.environ", "getenv(", "api_key", "BEGIN ", ".env")
MAX_PROD_FILES = 3


def git_names(diff_range: str) -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", diff_range], text=True
    )
    return [line.strip() for line in out.splitlines() if line.strip()]


def patch_text(diff_range: str) -> str:
    return subprocess.check_output(["git", "diff", diff_range], text=True)


def classify(diff_range: str, requested: set[str]) -> str:
    names = git_names(diff_range)
    body = patch_text(diff_range).lower()
    prod = [n for n in names if not Path(n).parts[0].startswith("test")]
    extra = [n for n in names if n not in requested and not n.startswith("test")]

    if any(h in n.lower() for n in names for h in ARCH_HINTS):
        return "LEAF_D_REWRITE_architecture_touch"
    if any(h in body for h in SECRET_HINTS):
        return "LEAF_A_DISCARD_secret_or_env_touch"
    if extra or len(prod) > MAX_PROD_FILES:
        return "LEAF_A_DISCARD_silent_expansion"
    if not names:
        return "LEAF_A_DISCARD_empty"
    if all(n.startswith(ALLOWED_PREFIXES) for n in names) and len(prod) <= 2:
        return "LEAF_C_LOCAL_allowlist"
    return "LEAF_B_QUARANTINE"


if __name__ == "__main__":
    if len(sys.argv) < 3:
        print("usage: classify_patch.py   [more files]")
        sys.exit(2)
    decision = classify(sys.argv[1], set(sys.argv[2:]))
    print(decision)

Run it against a generated branch, not against main:

git fetch origin
git checkout -B agent/try-1 origin/agent/try-1
python3 classify_patch.py main...HEAD src/billing/quote.py tests/test_quote.py

The printed leaf is a starting label. Override it when you have information the script cannot see, such as a compliance boundary or a frozen schema.

Leaf A — Discard and tighten

Scene. Prompt: “Add a quote_total(items) helper.” Diff: quote.py, logger.py, utils/retry.py, and a new requirements pin on a metrics SDK.

Why this leaf. Surface area exploded. The extra files are not tests. The metrics SDK implies network. Step 2 and Step 3 both fail.

Worked action.

1. Do not merge. Do not cherry-pick “just the helper” until you have re-read the helper in isolation; expansions often leak into the helper itself.
2. Record the extra paths in the prompt’s reject list.
3. Regenerate with an explicit file cap and an oracle name.

Task: add quote_total(items) in src/billing/quote.py only.
Do not create files. Do not edit requirements or logging.
Oracle: tests/test_quote.py::test_quote_total_cents must pass.
If the oracle needs a test change, edit that test file only.

What “done” looks like. A new diff with one production file, or a decision to write the helper by hand because the review budget is already spent.

Leaf B — Quarantine run

Scene. Prompt: “Parse CSV invoices in src/invoices/parse.py.” Diff: that file plus tests/test_parse.py. No auth, no Docker, no env reads. You do not want that parser executing against files in your home directory.

Why this leaf. Bounded surface, no secret touch, and an oracle exists. Isolation is the remaining requirement.

Worked action. Copy the branch into a throwaway directory or machine. Feed only fixture files. Run the oracle. Throw the machine state away.

# proposal workflow — run on a disposable host, not on a laptop with SSH keys loaded
mkdir -p /tmp/quarantine && cd /tmp/quarantine
git clone --depth 1 --branch agent/try-1 /path/to/local/mirror invoices
cd invoices
python -m venv .venv && . .venv/bin/activate
pip install -e '.[test]'
pytest tests/test_parse.py -q --fixtures-per-test

If the oracle needs a CSV, mount a fixture directory that contains no customer data. If the model added requests.get, the run still belongs on Leaf A, even if pytest is green: the classifier should have caught it, and the quarantine host should have no egress if you can help it.

A quarantine host can be a local container. It can also be a spare server that never sees production credentials. MonkeyCode is relevant on this leaf only: it currently offers free model access and a free server option, which is one way to keep generation and first-oracle runs off your workstation. Disclosure: This article was prepared as part of MonkeyCode’s product outreach. Treat both the models and the server as capacity that can change; verify current terms on the project before you plan around them. Do not place secrets, customer fixtures, or deploy keys on a shared free server.

What “done” looks like. Oracle green on the isolated copy, plus a human skim of the two files before allowlist apply onto a real branch.

Leaf C — Local allowlist apply

Scene. Prompt: “Extract cents(amount: str) -> int.” Diff: src/money.py and tests/test_money.py. Pure functions. No I/O. Classifier prints LEAF_C_LOCAL_allowlist.

Why this leaf. Isolation would still be nicer, but the blast radius is a two-file pure change and the oracle is local unit tests you already trust.

Worked action.

git checkout main
git checkout agent/try-1 -- src/money.py tests/test_money.py
git diff --cached --stat
pytest tests/test_money.py -q

Stop if --stat shows any other path. Stop if the test file grew assertions about logging, time, or HTTP. Those are expansion in disguise.

What “done” looks like. Two allowlisted files, oracle green, no other staged paths.

Leaf D — Rewrite

Scene. Prompt: “Make login less flaky.” Diff: auth/middleware.py, a session store swap, and a Dockerfile change to add Redis.

Why this leaf. Architecture-touching. Step 3’s “real network need” is present, and there is no dedicated staging path in the scene. Quarantine cannot validate session semantics with unit tests alone. Local apply on a developer laptop is how flaky login becomes an outage.

Worked action.

1. Freeze the public contract in an oracle first: status codes, cookie names, expiry, and failure bodies.
2. Split the work into patches the tree can route: store behind an interface, then middleware, then infra. Each patch re-enters at Step 1.
3. If the review budget is already gone, write the interface by hand. Cheap generation does not refund hours spent reading a Redis session rewrite you did not schedule.

# proposal: freeze the contract before any generated middleware lands
def test_login_failure_body_stable(client):
    res = client.post("/login", json={"user": "x", "password": "bad"})
    assert res.status_code == 401
    assert set(res.json().keys()) == {"error", "code"}

What “done” looks like. A smaller patch that re-enters Leaf B or C, or a human-written change. Not a fourteen-file “login fix.”

Limitations

The tree does not detect vulnerabilities, license issues, or subtle numeric drift. The classifier is string heuristics. It will miss a polished secret read and it will over-flag a comment that mentions .env.

Quarantine is not staging. A green oracle on a throwaway host does not mean production behavior. Free model access and a free server do not add an oracle; they only move generation and first execution off your laptop. If you have no contract tests, you do not have Leaf B. You have a remote place to watch the same untested code fail.

Do not use this approach when the repo holds regulated data, when the agent needs production-like secrets, or when the change is a public contract you cannot freeze in a test. Do not use it as a substitute for architecture review on auth, payments, or migrations. Teams that cannot add a fifteen-minute oracle should not scale cheap generation; they should shrink the task.

The routing bias toward discard will feel slow compared with “accept all files.” That slowness is the point. When patches are cheap, the scarce resource is attention. Spend it on oracles and surface-area caps, not on reading surprise Dockerfiles at 9:12 a.m.

If Leaf B is the leaf you keep landing on, verify whether an isolated server is actually isolated, then run the oracle there. MonkeyCode’s free model access and free server option are one place to try that isolation; confirm current availability before you schedule work against them.