The Reliability Retainer keeps a production agent accurate after launch: regression runs on every change and every model release, judge recalibration as your traffic shifts, prompt and tool fixes, and a monthly report on where quality stands. Document formats change, user behaviour drifts, models get deprecated. Something has to notice.
The golden set re-runs on every prompt, tool or code change and on every model release you consider, so accuracy is a tracked number over time, not a claim made once at launch.
The LLM judge is re-checked against fresh human labels as your traffic shifts, so the scores keep meaning what they meant at launch.
Failures from production and from the exception queue get fixed and folded back into the golden set, so coverage grows instead of plateauing.
Scores by task, regressions caught, fixes shipped, and what the next model release would change. Something you can hand to a board or a customer.
An agreed SLA for when something breaks, and a named engineer who already knows the system.
It is closer to operations than support. Most of the work is measurement and adaptation: re-running evals, recalibrating judges, absorbing new formats and new models.
You can, though the useful moment is at handover, while the golden set is current. Picking it up a year later usually starts with rebuilding the measurement that lapsed.
No. After an Agent Reliability Audit the harness is in place, and the retainer can run it on an agent your team built.
Entry · AI teams · 2–3 weeks
The Agent Reliability Audit is a fixed-scope, 2–3 week engagement that turns your production traces into an eval harness your team runs on every change. We sample your traces, classify how the agent actually fails, build a golden set of 150 to 300 cases, calibrate an LLM judge against your team’s labels, and wire the result into CI.
Entry · AI teams · 2–3 weeks
Model Migration is an eval-gated switch to a newer or cheaper model, scored side by side on your own traffic. In 2–3 weeks you get quality, cost and latency compared case by case on a golden set, routing and caching where they pay off, and a cutover plan with a rollback path, so the decision rests on measurements rather than a benchmark someone else ran.
Entry · Operations · 2 weeks
The Workflow Pilot is a two-week, fixed-scope engagement: we build a working extraction or decision pipeline for one workflow, run it on your real documents, and measure its accuracy on about 200 samples. Nobody should sign a build contract to find out whether an AI system can handle their paperwork.
Next step
Audit the agent you already run, or pilot one workflow on your own documents. Either way, you leave with a number you can check.