Model Migration is an eval-gated switch to a newer or cheaper model, scored side by side on your own traffic. In 2–3 weeks you get quality, cost and latency compared case by case on a golden set, routing and caching where they pay off, and a cutover plan with a rollback path, so the decision rests on measurements rather than a benchmark someone else ran.
Your current model scored on cases drawn from your own traffic. If you do not have a golden set, we build one first; it is part of the engagement.
Each candidate model scored on the same cases for quality, cost per task and latency, with the regressions listed case by case rather than averaged away.
Where the scorecard supports it, cheaper models handle the easy traffic and the expensive one handles the rest, with prompt and response caching where it holds up.
A staged switch, gated on the eval, with a tested path back to the old model if production disagrees with the scorecard.
We build one from your traces in the first week. It is included, and it stays in your repo for every migration after this one.
We do not promise a number before measuring. The scorecard reports cost per task before and after on your own traffic, and you decide with that in front of you.
Then the scorecard shows where, and the usual answer is routing: the cheaper model takes the cases it handles well and the rest stay where they are. If nothing clears the bar, you keep the current model and the evidence for why.
Every major model release raises the question again. With the harness in your repo, the next migration is a re-run rather than a project, and the Reliability Retainer can run it for you.
NDA before anything is shared. We work with scoped read access inside your cloud or on de-identified samples, nothing goes to a third-party eval platform, and our copies are deleted at handover. The security page has the detail.
Entry · AI teams · 2–3 weeks
The Agent Reliability Audit is a fixed-scope, 2–3 week engagement that turns your production traces into an eval harness your team runs on every change. We sample your traces, classify how the agent actually fails, build a golden set of 150 to 300 cases, calibrate an LLM judge against your team’s labels, and wire the result into CI.
Core · 4–10 weeks
The Production Agent Build ships the agent, its integrations and a human exception queue in 4–10 weeks, with evals and monitoring running from the first day rather than bolted on at the end. Most agent projects stall because nobody can prove the thing is right. We ship the proof alongside the system.
Ongoing · Monthly
The Reliability Retainer keeps a production agent accurate after launch: regression runs on every change and every model release, judge recalibration as your traffic shifts, prompt and tool fixes, and a monthly report on where quality stands. Document formats change, user behaviour drifts, models get deprecated. Something has to notice.
Next step
Audit the agent you already run, or pilot one workflow on your own documents. Either way, you leave with a number you can check.