Most AI systems are measured once, on launch day, by the people who built them. Then a prompt gets edited, a model gets deprecated, a carrier redesigns a form — and nobody finds out that accuracy moved.
The problem
A traditional bug announces itself. An AI system that has quietly dropped from 94% to 81% field accuracy does not — it keeps returning confident answers, and the wrong ones look exactly like the right ones. In back-office insurance work that difference shows up months later, in an E&O conversation or an audit.
The only defence is to measure continuously against a fixed set of known-correct answers. That is all an eval harness is, and it is remarkable how many vendors ship without one.
The mechanism
Built from your real submissions, policies, or loss runs — deliberately weighted toward the hard cases. The scanned fax, the manuscript form, the carrier whose layout changed last year.
Accuracy per field, not one number for the document. A system that nails the named insured and fumbles the deductible is not 90% right in any way that matters.
Prompt edits, model swaps, and code changes are scored before they ship. If accuracy drops below the agreed threshold, the change does not go out.
Broken out by document type and source, so a weak format surfaces instead of being averaged away by the easy cases.
In practice
An illustrative scoring run for a submission-intake system. The shape is what matters: per-field, per-segment, with a threshold that blocks the release.
| Field | Clean PDF | Scanned | Threshold | |
|---|---|---|---|---|
| Named insured | 99.1% | 96.4% | 95% | pass |
| Effective date | 98.7% | 94.2% | 95% | fail |
| Total insured value | 97.3% | 95.8% | 95% | pass |
| Deductible | 96.9% | 93.1% | 95% | fail |
Two fields fail on scanned documents. That is not a reason to abandon the system — it is a reason to route scanned effective dates and deductibles to a human until the number comes up. Without the harness you would not know which two fields, or that scans were the problem at all.
Questions
Smaller than people expect. One to two hundred labelled examples, chosen to cover the document types and edge cases you actually receive, catches most regressions. A thousand random samples that are all the clean case is worth less than eighty deliberately awkward ones.
We build the first version during the sprint and your team corrects it, because the person who does the work is the one who knows what the right answer is. After that it grows from the exceptions the system routes to humans.
The replacement is scored against the same test set before it goes anywhere near production. A model upgrade becomes a measured decision with a number attached rather than a forced migration you hope went fine.
It is the same instinct applied to a non-deterministic system. The difference is that the pass condition is statistical — accuracy above a threshold on a population — rather than a single assertion being true.
Related
Evals answer is it right. The audit trail answers how did it arrive at that. Regulated work needs both, which is why they ship together rather than as options.
Both are part of every production build, and the test set is yours to keep from the end of the sprint onward.
Next step
Then ask how they measured it, and when they last re-ran it. We would rather you asked us that question than took our word for anything.