The Agent Reliability Audit is a fixed-scope, 2–3 week engagement that turns your production traces into an eval harness your team runs on every change. We sample your traces, classify how the agent actually fails, build a golden set of 150 to 300 cases, calibrate an LLM judge against your team’s labels, and wire the result into CI.
The golden set, the gate in CI, and the failure taxonomy.
Every failure in a stratified sample of your traces, classified: wrong tool call, bad retrieval, invented field, instruction drift, and so on. Ranked by frequency and severity, so you know which fix buys the most.
150 to 300 cases drawn from real traces, weighted toward the hard cases and the incidents you already know about, with expected outcomes your engineers sign off.
Deterministic checks wherever a deterministic check can work. An LLM judge only where it cannot, calibrated against human labels, with the agreement rate reported in a calibration report.
The harness runs on every pull request in the CI you already use, with pass and fail thresholds per task. A regression blocks the merge instead of reaching users.
Where the agent stands today, task by task, and a scoped roadmap of the fixes that move the score. Plain code in your repo; adding a case is a pull request, not a call to us.
One workflow and 150 cases. Taxonomy, golden set, judge calibration and scorecard.
Up to three workflows, plus the CI integration and per-task thresholds.
Everything in Standard, an adversarial and safety suite, and 30 days of support after handover.
We start from your logs, or seed cases from your product docs and support tickets. Traces make it faster, not possible.
You do. The code, the golden set and the judge prompts live in your repo, in your stack, documented for your team.
Platforms store and run evals. We design the tests and calibrate the judges, and the harness runs on whichever platform you already use, or on none.
Only where a deterministic check cannot work, and only after the judge has been checked against your team’s human labels. An unvalidated judge just moves the guesswork one layer down.
You get a scoped fix roadmap. Your team can work through it, we can do the fixes as a scoped engagement, and the Reliability Retainer keeps the harness current as the agent and the models change.
NDA before anything is shared. We work with scoped read access inside your cloud or on de-identified samples, nothing goes to a third-party eval platform, and our copies are deleted at handover. The security page has the detail.
Entry · AI teams · 2–3 weeks
Model Migration is an eval-gated switch to a newer or cheaper model, scored side by side on your own traffic. In 2–3 weeks you get quality, cost and latency compared case by case on a golden set, routing and caching where they pay off, and a cutover plan with a rollback path, so the decision rests on measurements rather than a benchmark someone else ran.
Core · 4–10 weeks
The Production Agent Build ships the agent, its integrations and a human exception queue in 4–10 weeks, with evals and monitoring running from the first day rather than bolted on at the end. Most agent projects stall because nobody can prove the thing is right. We ship the proof alongside the system.
Ongoing · Monthly
The Reliability Retainer keeps a production agent accurate after launch: regression runs on every change and every model release, judge recalibration as your traffic shifts, prompt and tool fixes, and a monthly report on where quality stands. Document formats change, user behaviour drifts, models get deprecated. Something has to notice.
Next step
Audit the agent you already run, or pilot one workflow on your own documents. Either way, you leave with a number you can check.