Properties First, Fixtures Second, Flakes Never: A Test Contract for Agent Patches

An agent patch is a hypothesis. A test suite is the only evidence a reviewer gets. Most suites fail at that job in three reproducible ways: assertions the agent can reverse-engineer, fixtures that regenerate and drift, and flaky tests that flap between green and red without a single commit. This article is a three-layer contract that closes all three: property checks for invariants, hash-pinned fixtures for determinism, and a flake quarantine that runs before the agent ever sees the suite.

The evidence problem

assert(add(2, 2) == 4) is not evidence. It is a target. An agent that watches that failure can patch until the value matches, and the suite turns green even when the surrounding logic is wrong. More assertions do not fix this; they add more targets.

Tests only become evidence when the agent cannot predict what will be checked. That requires three changes at once.

Layer 1: Property checks turn values into relationships

The first layer replaces fixed expected values with invariants. A property check asks: does the output violate the module’s documented contract for any input? The agent cannot memorize the answer, because there is no single answer.

C++ sketch with invariants for a normalize_path function:

#include 
#include 
#include 

// Converts '' to '/', collapses duplicate '/', preserves trailing '/'
std::string normalize_path(const std::string& p);

// Invariant 1: no backslash survives
void check_no_backslash(const std::string& in) {
  auto out = normalize_path(in);
  assert(out.find('\') == std::string::npos);
}

// Invariant 2: normalization is idempotent
void check_idempotent(const std::string& in) {
  auto once  = normalize_path(in);
  auto twice = normalize_path(once);
  assert(once == twice);
}

// Invariant 3: '' and '/' variants converge to the same output
void check_variants_converge(const std::string& in) {
  auto with_slash = in;
  std::replace(with_slash.begin(), with_slash.end(), '\', '/');
  assert(normalize_path(in) == normalize_path(with_slash));
}

The generator is a loop with a frozen seed. The seed is not a detail; it is the determinism guarantee:

#include 

std::string random_path(std::mt19937& rng) {
  static constexpr char alphabet[] = {'a', 'b', '/', '\', '.', '-'};
  std::string s;
  for (int i = 0; i < 64; ++i)
    s += alphabet[rng() % (sizeof alphabet)];
  return s;
}

int main() {
  std::mt19937 rng(0x5EED);  // frozen seed -> reproducible runs
  for (int i = 0; i < 500; ++i) {
    auto input = random_path(rng);
    check_no_backslash(input);
    check_idempotent(input);
    check_variants_converge(input);
  }
  return 0;
}

Compile with assertions enabled (-UNDEBUG); a compiled-out assert is a fake gate. You do not need a property framework for this pattern — a loop and a seed are enough — and that matters, because the contract should survive in projects that cannot adopt new tooling. The invariants check relationships, not values: an agent that “fixes” a violation must change behavior, not adjust a constant.

Layer 2: Frozen fixtures turn drift into a visible diff

The second failure mode is fixture drift. Generate a fixture at test time and it regenerates differently on the next run; a behavior change hides inside a file that “updated itself.” The agent’s diff looks clean, the reviewer sees nothing, and the regression ships. The fix is a frozen manifest: inputs are checked in, outputs are pinned, and a hash is asserted before the suite runs. The guard is a shell check, cheap enough to run on every attempt:

{
  "fixture": "paths/edge_cases.txt",
  "sha256": "9f2c91d4e8a17b3c...",
  "frozen_at": "2026-08-29",
  "covers": [
    "trailing slash preserved",
    "empty path",
    "duplicate separators",
    "UNC prefix"
  ]
}

#!/usr/bin/env bash
# guard_fixtures.sh — fail when a fixture changes without a manifest update
set -euo pipefail
FIXTURE="paths/edge_cases.txt"
EXPECTED="9f2c91d4e8a17b3c..."
ACTUAL="$(sha256sum "$FIXTURE" | cut -d' ' -f1)"

if [[ "$ACTUAL" != "$EXPECTED" ]]; then
  echo "fixture drifted: $FIXTURE"
  echo "expected $EXPECTED, got $ACTUAL"
  echo "review the diff, then update the manifest deliberately"
  exit 1
fi

The rule is simple: a fixture changes only through a committed manifest update. An agent patch that touches a fixture produces a diff a human must read. If the change is correct, approve it in review. If the agent “fixed” the fixture to match broken behavior, the diff is the evidence.

Layer 3: Flake freeze before the agent starts

The third failure mode has a classic signature: failed, passed, failed, with no commit in between. A test that flaps on a clean base is not a bug report; it is noise. Left in the feedback loop, it pushes an agent to “fix” a function that was already correct.

The freeze procedure:

1. Reset to the base commit on a clean checkout.
2. Run the suite three times.
3. Any test that fails at least once during the sweep goes into flake_quarantine.txt.
4. Filter quarantined tests out of the agent’s feedback loop entirely.
5. A quarantined test returns only after ten consecutive clean runs on a fixed machine.

Sweep script:

#!/usr/bin/env bash
# freeze_flakes.sh — 3x sweep on the base commit
set -uo pipefail
: > all_failures.txt

for run in 1 2 3; do
  ctest --output-on-failure -j4 > "run_${run}.log" 2>&1
  grep -E 'Failed' "run_${run}.log" |
    sed -nE 's/.*Test +#([0-9]+): ([^ ]+).*/2/p' >> all_failures.txt
done

sort all_failures.txt | uniq -c | sort -rn

Interpretation: a count of 1 or 2 means the test is flaky; quarantine it. A count of 3 means the base commit is genuinely broken; fix that before spending agent cycles. The cost of the sweep is the price of trustworthy evidence. The sed regex assumes one-word test names; adapt it to your runner’s output.

Where free models and a free server fit

Disclosure: This article was prepared as part of MonkeyCode’s product outreach.

The three-layer contract is compute-hungry by design: three suite runs per task, plus hundreds of property iterations. MonkeyCode offers free model access and a free server option, which covers exactly this class of disposable work. Use the server for the flake sweep and the property loops so they never consume CI minutes; let the free models draft the first generator pass and a fixture manifest seed from your existing test names. The model proposes, a human freezes. That division is the whole point: the quarantine list and the fixture hashes stay under human control.

What each layer catches

Limitations and who should skip this

Property checks require an expressible invariant. Pure functions, parsers, and path utilities fit; UIs, visual output, and time-dependent systems do not. The fixture freeze is only as honest as its reviewers — if every fixture diff is approved unread, the freeze becomes theater. The flake quarantine is triage, not deletion: a quarantined race-condition test is still a bug, so schedule a human to investigate the list weekly.

Skip this contract if the suite already takes more than thirty minutes per run (the sweep becomes the bottleneck), if the environment cannot be pinned (quarantine needs a fixed machine), or if the patch blast radius is a one-off script. The overhead is only justified where agent evidence is the product.

The contract in five steps

1. Freeze the base: run the 3x sweep, quarantine anything that flaps.
2. Write property checks for the module under patch; freeze the seed.
3. Pin fixtures with hashes; commit the manifest and the guard.
4. Only now hand the suite to the agent.
5. Review the agent’s test changes as strictly as its code changes. A patch that rewrites code and tests in one commit is often a patch learning to fake evidence.

A test suite is evidence only when the agent cannot reverse-engineer it and the environment cannot corrupt it. Properties remove the target; fixtures remove the drift; the flake freeze removes the noise. The free tier in MonkeyCode makes the sweep cheap and the first draft fast. The freeze itself remains a human decision. Keep it that way.