Coding agents are trained against a snapshot of the world. So when you ask one to wire up an LLM call, it reaches for whatever model string was current at its training cutoff — and writes something like gpt-4-vision-preview into your code without checking whether it still exists, or will exist in 2 months.
The code compiles, tests pass, CI is green. Then one morning the provider retires that model, and your production calls start returning errors.
I kept falling into that gap, so I built a tool for it.
The moment that made me write it
Last week I ran it across my own repos. It found claude-sonnet-4-20250514 in three of my live apps — days before Anthropic retires that model on June 15. Nothing in my dependency tooling had flagged it, because a model ID isn’t a dependency. I’d have shipped straight into the outage.
What it does
arol-ai is a CLI that scans your codebase for third-party APIs, SDKs, and model strings, checks each one against a hand-verified database of vendor sunset dates, and tells you what breaks and when. It runs locally, reports exact file:line, and only fails your build on high-severity or imminent deprecations (configurable, so it won’t nag you about something retiring eight months out).
npx arol-ai scan
“Isn’t this just Dependabot / grep?”
Fair question – it’s the first one everyone has asked so far.
The difference is where these things live and what triggers them. Model IDs and API calls sit in your code as plain strings, not as versioned packages in your lockfile — so your dependency graph never sees them, and Dependabot can’t either. And the trigger isn’t a version bump or a CVE. It’s a physical date. claude-sonnet-4-20250514 doesn’t gradually go “out of date” — it works perfectly right up until June 15, then your calls start erroring on a schedule you never wrote down anywhere.
Coverage
Today it covers OpenAI, Anthropic, and Google models and APIs, across JS/TS and Python codebases. Plus some other common infra providers like Vercel (this ain’t just for LLMs). Every sunset date comes from the vendor’s own changelog — if I can’t source a date, I don’t ship it.
Honest caveats (it’s early)
Right now arol flags references to deprecated models. It doesn’t yet tell a live API call apart from a model ID sitting in a config table or a test fixture, so if you point it at a big library you’ll see some noise. Call-site-aware scanning is what I’m building for early next week.
Local scans are free and will stay free.
Try it / tell me what breaks
I’d love feedback — especially false positives, and which providers or frameworks you’d want covered next. Run it on a repo and tell me what it catches (or wrongly catches):
npx arol-ai scan
- Site: https://arol.ai
- GitHub: https://github.com/benminor/arol
