Last year I renamed a field in an API response. customer_id became customerId, matching the rest of the codebase. Every test passed. Review approved it in about four minutes.
Three days later a mobile team filed a P1. Their client had been reading customer_id for eighteen months.
Nothing in the pipeline was broken. The tests tested our code, and our code was fine. What broke was the contract between us and someone else — and nothing in CI was looking at that.
I’ve spent the past year building tooling around this problem, and I used most of the existing tools along the way. This is what I learned about where each one actually wins.
Disclosure before we start: I’m the author of SpecShield, one of the tools below. I’ll be straight about where the free alternatives beat it, because for a lot of teams they will. If all you need is what oasdiff does, use oasdiff — it’s excellent and it costs nothing.
“Contract testing” is four different jobs
Most of the confusion in this space comes from one phrase covering four unrelated problems. Once you separate them, tool choice gets obvious.
Job 1 — Did this change break anything at all?
Diff two OpenAPI specs, classify what changed, flag what’s backwards-incompatible. Removed endpoints, narrowed types, newly-required request fields.
Job 2 — Is this spec any good?
Linting. Missing descriptions, inconsistent casing, no operationId, auth described in ways that will bite you later.
Job 3 — Will this break a specific consumer I know about?
Different question entirely. Removing an endpoint nobody calls is free. Removing one that three teams depend on is an outage. This needs knowledge of who consumes what.
Job 4 — Should this deploy be blocked?
The organisational one. Someone has to decide that a finding stops a merge, and someone has to be able to override it without turning the whole check off.
Most tools do one or two of these well. Nobody does all four, including me.
Job 1: breaking-change detection — oasdiff wins
oasdiff is open source, has several hundred breaking-change checks, ships a CLI and a GitHub Action, and has a hosted web diff where you paste two specs.
oasdiff breaking spec-v1.yaml spec-v2.yaml
It’s the reference implementation. If your requirement is “tell me when a PR breaks the spec,” install it this afternoon and skip the rest of this post.
I’ll say the uncomfortable part plainly, since it’s the honest framing: breaking-change detection is commoditised. It’s table stakes. Nobody should pay for it as a standalone feature, mine included.
Job 2: linting — Spectral and vacuum
Spectral is the de facto standard, and vacuum is a much faster Go implementation that reads the same ruleset format.
extends: [[spectral:oas, recommended]]
rules:
operation-operationId: error
operation-description: warn
Both free. Both good. Stoplight, Apigee API hub and Azure API Center all bundle Spectral under the hood, so if you’re on one of those you already have this.
The gap isn’t the linter. It’s what happens to the output. spectral lint prints a wall of warnings, someone puts it in CI, it fails on 200 findings the first day, and by week three it’s || true. I’ve watched this happen at two companies.
Job 3: consumer-aware verification — PactFlow, and this is where money lives
Pact and PactFlow solve the real version. Consumers publish what they actually use, providers publish what they offer, and a broker tells you whether a given pair is compatible.
can-i-deploy is the piece worth stealing conceptually:
pact-broker can-i-deploy --pacticipant payments-api
--version abc123 --to-environment production
That’s not “did the spec change.” It’s “will this specific build break a consumer that is live right now.”
PactFlow is the mature product here and starts around $99/month. If you want full consumer-driven contract testing with the Pact DSL and runtime verification, they go deeper than anything spec-only can — including mine. Spec-to-spec comparison can’t tell you a consumer reads a field it never declared. Runtime pact verification can.
Worth knowing: Optic was archived in January 2026. If you’re running it, it’s unmaintained. That surprised a few people I’ve spoken to.
Job 4: the gate — the part I think is underserved
Every tool above produces findings. Very few help you turn findings into a decision that survives contact with a deadline.
This is the gap I built SpecShield for, so weigh what follows accordingly.
A score, not a wall of warnings
Governance findings get weighted by severity into a 0–100 score and an A–F grade:
specshield govern openapi.yaml --min-score 80
Any error-severity finding caps the grade at C, on the theory that an API with an unresolved error is not an “A” API no matter how many nice descriptions it has. A threshold is something a team can agree on. “Zero warnings” is something a team agrees on once and then abandons.
Three rule packs ship built in:
| Pack | Rules | For |
|---|---|---|
recommended |
20 | Design consistency — operationId, descriptions, kebab-case paths, camelCase properties, array and string bounds |
owasp |
4 | HTTPS-only servers, security field defined, no credentials in query strings, no HTTP Basic |
minimal |
3 | The highest-signal rules only, for adding a gate to a legacy spec without drowning |
Waivers, so the gate survives
This is the feature I’d argue matters most, and it’s the least exciting to demo.
When a gate blocks something at 6pm on a Friday, the team has two options: fix it, or disable the check. Everyone disables the check. Then it never comes back on.
So findings can be waived at the org level, with the waiver recorded and revocable:
specshield govern openapi.yaml --org acme --min-score 80
The waiver applies to the verdict, the rule stays on for everyone else, and there’s a record of who accepted what. The gate stays credible because there’s a pressure valve that isn’t “delete the CI step.”
Rulesets that aren’t a lock-in trap
Custom rulesets are Spectral-format YAML. There’s a portability validator that actively rejects engine-specific constructs — vacuum-only extends, vendor rule-id prefixes, engine-specific functions.
That’s deliberate. A ruleset you write should run somewhere else if you leave. Governance rules encode institutional knowledge, and holding that hostage is a bad way to keep customers.
Output CI already understands
specshield govern openapi.yaml --output report.sarif
SARIF, JUnit, HTML and JSON. SARIF matters most — GitHub ingests it natively, so findings show up as annotations on the diff instead of buried in log output nobody opens.
The deploy gate
specshield bdct publish-provider --spec openapi.yaml --env production
specshield bdct can-i-deploy --provider payments-api --env production
Consumers register the contract they depend on. Before a deploy, you ask whether it’s safe. Same shape as PactFlow’s, built on OpenAPI rather than the Pact DSL — lighter to adopt, less powerful at runtime. That’s a genuine trade, not a free win.
Where it runs
The same engine is exposed through a CLI, a GitHub Action, a GitHub App that posts a check run and a sticky PR comment, an IntelliJ plugin, and an MCP server so coding agents can run the gate before opening a PR.
One thing worth stating clearly: it’s fully deterministic. No LLM anywhere in the analysis. The same spec always produces the same score, and every deducted point traces to a named rule. In a market currently bolting language models onto linters, “explainable and reproducible” felt like the more useful property for something that blocks deploys.
The comparison, honestly
| Breaking changes | Linting | Consumer-aware | Deploy gate | Waivers | Cost | |
|---|---|---|---|---|---|---|
| oasdiff | ✅ Best in class | ❌ | ❌ | Partial | ❌ | Free |
| Spectral / vacuum | ❌ | ✅ Standard | ❌ | ❌ | ❌ | Free |
| PactFlow | ✅ | Partial | ✅ Deepest | ✅ | Partial | ~$99/mo+ |
| Bump.sh | ✅ | Partial | ❌ | ❌ | ❌ | ~$50/mo+ |
| Optic | ✅ | ✅ | ❌ | ❌ | ❌ | Archived |
| SpecShield | ✅ | ✅ | ✅ | ✅ | ✅ | Free tier |
Reading that table fairly:
- oasdiff beats me at raw breaking-change detection. More checks, longer track record, larger community.
- PactFlow beats me at consumer-driven testing. Runtime verification catches things spec comparison structurally cannot.
- Spectral has a far bigger ruleset ecosystem than my three packs.
- Apigee and Azure API Center aren’t competitors at all — they’re runtime API management. Different layer, and complementary.
Where I think the combination earns its place: one gate that does breaking changes and governance, with a threshold and a waiver workflow, wired into the PR rather than a separate dashboard.
If you don’t need the gate, the free tools are genuinely enough. I’d rather say that than have you find out in week two.
What I’d actually recommend
Just starting? oasdiff in CI. Today. Takes twenty minutes and catches the majority of what hurts.
Spec quality drifting across teams? Add Spectral or vacuum. Start with three rules, not thirty.
Real consumers, real runtime coupling, budget? PactFlow. It’s the mature answer.
Want one gate, a score you can threshold, and a waiver trail? That’s the gap I built for — specshield.io, free tier, CLI is npm i -g specshield.
Still on Optic? It’s been archived since January. Please migrate to something.
The lesson from my customerId incident wasn’t that we needed more tests. We had plenty. It was that nothing in the pipeline knew a contract existed, or who was on the other end of it.
Whichever tool you pick, pick one that does.
What are you using for this? Genuinely curious whether people are gating on contracts yet or still finding out in production like I did.
