How do you actually test an AI agent? Not “does it respond,” but: does it
route to the right tool, chain calls correctly, recover from failure, resist
prompt injection, and stay within cost/latency budget?
I spent weeks working through this on a running agent, and open-sourced the
entire methodology — framework-agnostic, so it applies regardless of your
language, runtime, or toolset.
What’s inside
• 61-source benchmark map — BFCL, GAIA, τ-bench, SWE-bench, WebArena,
AgentDojo, LongMemEval and more, categorized by what they actually measure
• 58 universal test blocks across 7 tiers (L1–L4, Error Recovery,
Multi-Turn, Security). Each block = a tool-agnostic capability definition +
a concrete reference implementation
• Full OWASP Top 10 for Agentic Applications 2026 (ASI01–ASI10) mapped to
6 universal security test blocks
• Evaluation methodology — LLM-as-Judge biases, pass@k vs pass^k,
trajectory vs end-state, observability (OpenTelemetry GenAI), automated
red-teaming (garak, PyRIT, DeepTeam)
• Regulatory alignment — NIST AI RMF, MITRE ATLAS, EU AI Act, ISO/IEC 42001
How to use it
Take Part II, replace the reference-implementation fields with your own agent’s
tool names and expected outputs. The universal capability definitions need no
changes. Blank templates are included.
PheronAgent (a macOS agent with 50+ native/MCP tools) is included as a real
reference case study — but the methodology is the product, not the agent.
No marketing narrative: STORY.md documents the real bugs, real test runs, and
real corrections that shaped each version.
Docs are CC BY 4.0, templates are MIT. Issues and PRs welcome.
