Run and Compare AI Evaluations with a CLI for Developers and Coding Agents

TL;DR: This walkthrough shows how developers and coding agents can use Quantiles, an open-source AI evaluation platform licensed under Apache 2.0, to quickly run, analyze, and compare AI evaluations locally. We’ll use the SimpleQA Verified benchmark as an example throughout this post, letting you follow the commands, inspect the evaluation results, and configure your own model for the same workflow.

Running an AI evaluation is rarely as simple as sending prompts to a model. Developers must connect datasets, model APIs, scoring logic, result storage, and comparison tooling before they can answer a basic question: did the system get better? When those pieces are spread across scripts, notebooks, and logs, every rerun becomes harder to reproduce and diagnose. A score alone cannot reveal whether the model changed or whether the dataset, prompt, scorer, or sample set changed with it.

Quickstart: Run an example benchmark

The Quantiles CLI is called qt on the command line. A simple curl ... | bash command supports macOS and Linux on X86-64 and Arm64 systems. First, use it to install the CLI:

curl -fsSL https://cli.quantiles.io/install.sh | bash

If you don’t want to run code directly sourced from the internet, see the install.sh source code first.

Next, let’s run a built-in benchmark from start to finish using a single command. SimpleQA Verified is a 1,000-prompt benchmark created by Google DeepMind and Google Research. It re-curates questions from OpenAI’s SimpleQA benchmark to reduce problems such as incorrect labels, topical bias, redundant questions, and ambiguous source evidence. Each example includes a short factual question in problem, its reference answer, topic and answer-type metadata, and supporting URLs.

Use the following command to run simpleqa-verified using the built-in Quantiles demo model, which doesn’t incur any usage charges:

qt run simpleqa-verified

Results from the demo model are intended only to demonstrate the evaluation workflow because its outputs are generated randomly. See the configuration section to run evaluations with your own model.

The current Quantiles integration reads problem and answer columns from the quantiles/simpleqa-verified dataset, which is a fork of the canonical Google dataset. The benchmark remains the work of the original authors.

The above qt run command embeds each response and reference answer using a local embedding model powered by fastembed, and then compares the two using cosine similarity. It then records a sample-level similarity score and aggregates the score distribution. This flow exercises dataset loading, sample execution, scoring, aggregation, and evaluation metadata and metrics storage, without using an LLM provider. It does not reproduce the benchmark’s canonical GPT-4.1 autorater, so does not report the resulting similarity metrics as SimpleQA Verified model performance.

When you’re ready to evaluate LLMs, refer to the model configuration documentation section below for detailed setup instructions.

Quantiles calculates and displays the aggregate metrics after all samples have been evaluated and reports the evaluation name, status, timestamps, duration, workflow input and output, error state, and aggregate metrics.

$ qt run simpleqa-verified

Created run 1
simpleqa-verified: 100%|████████████████████| 1000/1000 [00:18<00:00, 1195.52it/s]
Run 1 completed successfully in 19.21s

Aggregate metrics
 METRIC                VALUE     UNIT
 max_similarity        0.8351    -
 mean_similarity       0.5647    -
 median_similarity     0.5589    -
 min_similarity        0.4497    -
 p95_similarity        0.6359    -
 p99_similarity        0.6734    -
 stdev_similarity      0.0396    -
 variance_similarity   0.0015    -

Run `qt show 1 --json` for sample-level details.

Inspect and analyze sample-level results

Each completed evaluation reports its run_id. If the ID is no longer visible, use the following command to list previous runs and their IDs:

qt list

Use the following command to display the sample-level details for a specific run. Here, run_id 1 refers to the SimpleQA Verified evaluation completed above.

qt show 1 --json

Use the --json flag to retrieve sample-level results and execution details as structured, machine-readable output. The JSON document has three top-level fields: run, metrics, and samples. Depending on the recorded step, a sample might include its step key, status, input hash, timestamps, stored output, error, and associated metrics.

Compare two benchmark runs

Many system changes can affect model behavior, including changes to prompts, datasets, code, infrastructure, and the model itself. Comparing two runs makes it possible to measure and inspect the effects of a change.

Run the same simpleqa-verified evaluation a second time and retain both run IDs:

qt run simpleqa-verified

In this example, the two SimpleQA Verified runs have run_id values of 1 and 2:

qt compare 1 2

Comparing runs 1 and 2
                       Run 1              Run 2              Delta
Eval                   simpleqa-verified  simpleqa-verified  SAME
Status                 COMPLETE           COMPLETE
Duration               23.201s            18.720s            -4.481s
Model                  demo-builtin       demo-builtin       SAME
max_similarity         0.7409             0.7618             +0.0209
mean_similarity        0.5634             0.5647             +0.0013
median_similarity      0.56               0.56               -0.000041
min_similarity         0.4501             0.4341             -0.0161
p95_similarity         0.6387             0.6397             +0.001
p99_similarity         0.6779             0.6798             +0.0018
stdev_similarity       0.0409             0.0407             -0.000228
variance_similarity    0.0017             0.0017             -0.000019

Run 'qt compare 1 2 --json' for sample-level details

qt compare exits with status 1 when any checked input, output, step, or aggregate metric differs, and 0 when the runs match, which can be useful for scripts or CI jobs. It’s important to remember, however, that a 1 status means “different,” and doesn’t necessarily indicate that the model got worse. Even a metric improvement counts as a difference. Make sure to apply your own release policy to the JSON output and the status code.

Customize the benchmark

So far, the examples have used the default demo model and the full evaluation dataset. You can instead configure a model from a supported provider, such as OpenAI or Anthropic, and set a sample limit when you do not need to evaluate all 1,000 prompts. Provider-backed evaluations send prompts to an external service and may incur usage charges, so start with a small smoke test.

--input flag

The quickest way to customize a built-in benchmark such as SimpleQA Verified is to use --input to override the model or sample limit. These overrides apply only to the current run and the value passed must be a JSON object. To reuse the settings in future SimpleQA Verified runs, add them to quantiles.toml configuration file.

For example, this command limits SimpleQA Verified to 10 samples while still using the built-in demo model:

qt run simpleqa-verified --input '{"limit":10}'

To evaluate the same subset with a specific model, such as one from OpenAI, add the model to the JSON object and ensure that OPENAI_API_KEY is already set:

qt run simpleqa-verified --input '{"limit":10,"model":"openai:gpt-5.6-luna"}'

Configuration file

You can use a configuration file when you want the same model and sample settings to apply across repeated runs. To do this, create exactly one of either quantiles.toml or .quantiles.toml in the directory where you run qt (the CLI will throw an error if it finds both).

The below quantiles.toml configuration runs the first 10 SimpleQA Verified samples with an OpenAI model:

[benchmarks.simpleqa-verified]
samples = 10
model = "openai:gpt-5.6-luna"

Omit the samples key from configuration if you intend to run the evaluation over its complete dataset.

Quantiles loads the matching benchmark section automatically when you run:

qt run simpleqa-verified

Choose a model and configure its API key

Quantiles currently supports four model provider prefixes. It forwards the text after the prefix as the provider’s model ID, so the available models depend on the provider’s current catalog and your account access. The repository examples below show identifiers that work with the current configuration format.

Supported providers and model ID formats

Cloudflare includes models from families such as OpenAI’s gpt-oss, Llama, Mistral, Gemma, DeepSeek, Qwen, and GLM. Use the Cloudflare model catalog to choose an exact model ID because availability and access can change independently of Quantiles.

Keep API keys out of quantiles.toml, --input JSON, and source control. Quantiles reads provider credentials from environment variables, so set the variable listed in the table for the selected provider.

For the OpenAI examples below, export the key in the same terminal session where you will run your qt commands:

export OPENAI_API_KEY=""

Credentials are configured the same way whether you pass the model through a configuration file or the --input flag.

Use a coding agent for evaluation workflows

Quantiles provides an open-source agent skill that gives coding agents, such as Codex or Claude Code, reusable instructions for running evaluations, inspecting sample-level results, comparing runs, and recovering interrupted work with the qt CLI. Run the agent from your project root so it can find your Quantiles configuration and local run history.

First, ask your coding agent to install the skill:

Please install the Quantiles skill at github.com/quantiles-evals/skill

Then prompt your agent to run the SimpleQA Verified benchmark with the demo model:

Run the simpleqa-verified benchmark and summarize the results.

When you are ready to evaluate your model, replace the placeholder in the following prompt and run a small smoke test to verify the model configuration:

Run 10 samples of the simpleqa-verified benchmark using . Confirm that the required provider credentials are available without showing their values; if they are not, stop and tell me what is missing. When finished, summarize the results. Do not run the evaluation until I confirm.

After you have two comparable runs, ask the agent to analyze the change:

Compare the two most recent runs for 'simpleqa-verified'. Summarize the aggregate metrics, sample-level results, failures, and any notable errors. Identify the highest-impact issues to review first, and recommend specific next steps.

Limitations and reproducibility

• The built-in demo model generates random, unseeded text. Run IDs, timings, responses, and similarity values will differ between runs, and none of these results should be interpreted as evidence of model quality.
• SimpleQA Verified is designed to test answers produced without web search or external tools. When evaluating a model, disable tool access so the run measures the intended task. A coding agent can orchestrate the evaluation workflow, but it should not use tools to answer the benchmark questions.
• The first run with a new dataset requires network access to load dataset metadata and any uncached rows. The qt CLI uses the default Hugging Face revision, so be sure to record the dataset repository, configuration, split, and revision in your configuration file when strict reproduction matters.
• Model evaluations send prompts to an external service and may incur usage charges. Provider availability and behavior can also change independently of Quantiles.

Conclusion

The Quantiles stack provides a structured, local-first system for running, analyzing, comparing, and reproducing AI evaluations. You can use the built-in demo model to verify that your workflow is configured correctly, then switch to a model from a supported provider (e.g., OpenAI or Anthropic) with a fixed configuration to evaluate real model behavior consistently. Since running evaluations, analyzing results, and comparing runs all use the same CLI, the entire process can be carried out through your preferred coding agent.

If you’d like your benchmark included in Quantiles so others can run it with a single command, submit a request through the Quantiles issue tracker.

Documentation and references

• Quantiles – overview of the local-first AI evaluation platform
• Quantiles documentation – installation, configuration, CLI, SDK, and evaluation workflow guides
• Quantiles GitHub repository – source code, examples, releases, and issue tracker
• Quantiles agent skill repository – reusable instructions and installation files for coding-agent evaluation workflows