Regression evals for your agent, gating every merge.

Most seed and Series A teams ship agent changes on manual spot checks: a prompt edit looks fine in three tries and quietly breaks a slice of real traffic. In two weeks we install the harness that catches it: a labeled set from your own traces, a scoring run in CI that blocks the merge, and a taxonomy of how your agent actually fails.

Price
$8k–$15k
Fixed before kickoff, set by how many agent tasks and tools the harness covers.
Duration
2 weeks
Book a scoping call →How we prove it works →
What you get

Inside the Eval Harness Install

A labeled set from your traffic

Sampled from your production traces and weighted toward the hard cases and past incidents, with expected outcomes your engineers sign off. Typically one to three hundred cases, sized to the agent’s task surface.

A CI-gated scoring run

Runs on every pull request in the CI you already use. Per-task scores against agreed thresholds, so a regression blocks the merge instead of reaching users.

A failure taxonomy

Every failing case classified: wrong tool call, bad retrieval, invented field, instruction drift, and so on. You see which failure modes dominate and what fixing each one would buy.

A harness your team owns

Plain code in your repo, documented, in your stack. Adding a case is a pull request, not a call to us.

How it runs

The shape of the engagement

Days 1–2
Walk the agent with the engineers who own it. Sample production traces and agree what a correct outcome is for each task.
Days 3–6
Label the set with your team, weighting toward edge cases and the incidents you already know about.
Days 7–9
Build the scoring run: deterministic checks wherever possible, model-graded checks only where they are validated against your human labels.
Day 10
Wire it into CI, set thresholds, classify the failures into the taxonomy, and hand over.
Fit

This is for you if

  • You ship an LLM agent or RAG feature to real users, and changes are checked by hand
  • A prompt edit or model upgrade has broken something you heard about from a customer first
  • You have production traces or logs we can sample from

It is not, if

  • You have no users yet, so there is no real traffic to label
  • You already maintain an eval suite in CI and want more coverage rather than the foundation
  • You want a framework recommendation rather than a working harness
Questions

Before you book

Which stacks do you work with?

Whatever you run: the OpenAI or Anthropic APIs directly, LangChain or LangGraph, or your own orchestration. The harness is code in your repo, not a platform you have to adopt.

Do you use LLM-as-judge?

Only where a deterministic check cannot work, and only after the judge has been checked against your team’s human labels. An unvalidated judge just moves the guesswork one layer down.

What decides $8k versus $15k?

How many distinct agent tasks and tools the harness covers. We fix the price on a 30-minute call before kickoff, and it does not move after that.

How does our data stay private?

NDA first. We work inside your environment or on redacted traces, and nothing is sent to a third-party eval platform.

Going further

When you need more than evals

02 · $20k–$60k

Production AI Agent Build

Most agent projects stall because nobody can prove the thing is right. We ship the proof alongside the system.

03 · Scoped

AgentOps Retainer

The AgentOps Retainer is our monthly operations service for shipped AI systems: scheduled eval regression runs, drift monitoring, and model upgrades tested against your own test set. Document formats change, suppliers redesign their templates, models get deprecated. Something has to notice.

Next step

Start with one workflow.

A week, a fixed fee, and a measured answer on your own documents. If it will not work, you find out for $2,500.

Book a 30-min call →The $2,500 sprint →