AI / 07 of 08
Eval Sets
Freeze every production failure alongside the answer it should have given, so that every future change to an AI pipeline re-sits a graded exam before it ships.
How it runs
Trigger
A pipeline gets it wrong
Step 01
Freeze the case as it arrived
Step 02
Write the true answer by hand
Step 03
The case joins the set
Step 04
Someone proposes a change
Step 05
Re-run the whole set
Grade gate
Score against the baseline
Outcome
Ship or block
What it is
An eval set is a frozen collection of your pipeline's real past mistakes, each stored with the answer it should have given.
You need this if changing a prompt is a leap of faith, and the way you find out you broke something is a client telling you.
Every silent regression spends trust you took months to earn.
If you already test your automations, this is the version that tests judgement rather than plumbing.
What it does for you
Every prompt change is graded against real past failures before it ships
so fixing one case stops quietly breaking three others.
Regressions surface in the run instead of in a client's inbox
which is the difference between a fix and an apology.
The set only ever grows
so the system gets harder to break the longer it runs.
Built and running
Built internally for the business. Runs against the qualification gates on the prospect intake pipeline, so a change to the gate logic has to beat the previous score before it is allowed through.
How to build it
- Pick one pipeline where a wrong answer actually costs you something
- Every time it gets something wrong, save the exact input and write the answer it should have given
- Stop at twenty cases before you build anything, because the set is only useful once it covers the ways it really fails
- Write a scorer that grades a run against the set and prints one number
- Record today's number as the baseline, then refuse to ship any change that scores below it
- Add every new production failure to the set so that the exam gets harder over time
Want this one built into your business?
Thirty minutes, no deck. We map what you run today and tell you what it would take.