How to Build an AI Evaluation Set That Catches Regressions

Choose the change the set must judge

Suppose a support team wants to replace its current routing prompt. The model assigns each message to billing, access, bug, or needs_review and writes a short summary. The release question is specific: does the new prompt fix known routing mistakes without losing invoice numbers, closing a current subscription term, or claiming that an action has already happened?

Save the current prompt as the baseline. Name the candidate change, such as a new rule for distinguishing a billing question from a cancellation request. Write the release rules before looking at candidate results. In this example, an unsupported claim that a refund was issued blocks release. A mislabeled routine question needs review against the baseline. A clearer summary is useful only after those checks pass.

These are example rules for the support workflow, not universal thresholds. A team evaluating a draft email or a retrieval system will need different failure definitions. The useful starting question is what mistake would make you keep the current version.

Collect cases from actual failure modes

Look at corrected outputs, reviewer comments, escalations, and policy exceptions. For each failure, keep the input that caused it, what went wrong, and the expected behavior. Remove private data before sending a case to an external model. A synthetic replacement should preserve the decision boundary: changing a real invoice number to INV-1042 is fine; replacing a confusing cancellation request with an easy one is not.

If the workflow has no history yet, ask a domain owner to write ordinary requests and policy-boundary cases from the task specification. Mark them as expert-written rather than production failures, and have another reviewer approve the expected outcome. Add reviewed real failures as they arrive; a generated example should not silently become evidence of how often a problem occurs.

Do not fill the set with twelve near-identical examples of a single category. The downloadable 12-case fixture contains billing, access, bug, and needs_review requests. It includes missing information and instructions such as “cancel renewal, but do not close the current term.” It is a compact teaching fixture, not a sample of production frequency or a complete release set.

Each row is a case-design prompt, not a claim that the published 12 cases cover every situation. Select slices that your system will encounter, add high-consequence exceptions, and note which slices are still missing. OpenAI's evaluation guidance likewise recommends typical, edge, and adversarial cases rather than relying on one convenient sample.

Write an expected result a grader can inspect

For every case, record a stable ID, the complete input, an expected label or allowed outcomes, details that must survive, claims that must not be made, and why the case belongs in the set. Also keep a separate case register with its source, slice, severity, reviewer, and whether it is in the development set or holdout. The downloadable grader reads the ID, expected label, required fragments, and forbidden phrases shown below. It does not use the request text while grading, or read severity, split, and reviewer metadata.

{
  "id": "E01",
  "request": "I was charged twice for invoice INV-1042.",
  "expected_label": "billing",
  "must_preserve": ["INV-1042"],
  "must_not_claim": ["refund issued"]
}

The case register can be a spreadsheet; it does not have to be part of the grader's JSON input. For E07, one row might read: id=E07; source=synthetic policy example; slice=renewal versus current term; severity=release blocker; split=development; reviewer=support policy owner; reason=the model must not turn a request into a completed cancellation. Use source=reviewed customer escalation only when a real escalation supplied the case. Keep the reviewer's decision and date with the row so a later change to the expected answer is traceable.

Have a person who knows the support policy approve the expected label. If two reviewers disagree, resolve the policy question before treating either answer as ground truth. For a request that really could mean two things, needs_review may be the correct outcome. Record the reason; otherwise a later policy change can look like a model regression when the expected answer actually changed.

Keep development cases visible while editing prompts, but leave a holdout unseen until the candidate is chosen. All 12 downloadable cases are public, so they cannot serve as your private holdout. Do not quietly move a troublesome holdout case into development after seeing its result. Version the input, expected answer, and grader together when a policy changes. A changed expected answer should have its own review record.

Match the grader to the claim

Use exact checks for enum labels, stable IDs, JSON fields, and fixed output contracts. Use a human rubric or a model grader checked against human decisions for meaning that can be expressed in several ways. OpenAI documents several grader types; none makes a weak criterion sound merely by being more sophisticated.

The downloadable Python grader is deliberately literal. It checks the label, searches summary and evidence for required fragments and forbidden phrases, and rejects missing, duplicate, unexpected, or type-changed fields and IDs before scoring. It is useful for catching contract failures and known phrases. It cannot decide whether a paraphrase means that a refund happened. It can flag “no refund issued” because that sentence contains “refund issued,” or pass “a refund was completed” because those words differ. It can also pass an output whose summary has the wrong invoice number if the correct number appears elsewhere in evidence. If the summary is the field a worker uses, check that field specifically and review the sentence around the value.

For E07, give reviewers a short meaning-based rubric alongside the literal check. A passing summary says the customer wants renewal canceled while the current term remains open. “The subscription has been canceled” fails because it reports an action that the input only requested and loses the current-term boundary. “Cancellation requested” needs review because it leaves unclear what will happen to the current term. Apply the same rubric to paraphrases, and compare a model grader's decisions with these human labels before using it as a release gate.

Before trusting any grader, give it a clear pass, a clear fail, a paraphrased failure, a negated phrase, and a malformed record. Check false passes and false failures with a qualified reviewer. If a hard-stop rule depends on semantic judgment, do not let a string search be the only release gate. Store the grader's version and the review decisions alongside the run.

Run the same cases on the baseline and candidate

Freeze the prompt or workflow version, model identifier, output schema, tool permissions, retrieval snapshot, sampling settings where available, and test date. Keep the attempt count the same for both versions. A change to the prompt and the source documents at once makes the cause of a score difference unclear. For an API workflow, capture raw outputs with one record per case ID. The published observed file shows the required shape: each record has exactly id, label, summary, and evidence.

Download the expected cases, grader, and observed file into one directory. The direct command below checks the retained output. The separate fixture replay runner runs that successful check plus malformed-JSON, duplicate-ID, numeric-ID, and extra-field mutations; it tests the grading pipeline, not a new model candidate.

python3 ai-eval-regression-kit.py ai-eval-expected.json ai-eval-observed.json > observed-report.json
python3 advanced-ai-replay.py eval > replay-report.json

The first report should contain "valid_input": true, "passed": 12, and "total": 12. The replay report should contain "valid": true. A failed case or invalid input makes the grader exit nonzero. In an invalid-input report, the score fields are null and results is empty; do not treat it as a low but valid score.

For your own change, save the two raw outputs as baseline.json and candidate.json, with the same case IDs. Run the grader on each. A nonzero exit is expected when a valid run contains a failing case, so inspect both JSON reports rather than stopping at the shell status.

python3 ai-eval-regression-kit.py ai-eval-expected.json baseline.json > baseline-report.json
python3 ai-eval-regression-kit.py ai-eval-expected.json candidate.json > candidate-report.json

This grader produces one report per output file. It does not generate either model output or compare the reports for you. The following standard-library snippet lists newly failing and newly fixed case IDs. It stops if either report represents invalid input.

python3 - <<'PY'
import json

def results(path):
    with open(path, encoding="utf-8") as file:
        report = json.load(file)
    if not report["valid_input"]:
        raise SystemExit(f"Fix invalid input in {path} before comparing")
    return {row["id"]: row["passed"] for row in report["results"]}

old = results("baseline-report.json")
new = results("candidate-report.json")
print("New failures:", sorted(case for case in old if old[case] and not new[case]))
print("Fixed:", sorted(case for case in old if not old[case] and new[case]))
PY

Open the raw output for every changed case. “Label mismatch” tells you where to look, but not whether the candidate, expected answer, or policy is wrong. Use the case register to identify critical failures and slice-level changes; the snippet does not make the release decision for you.

Make the release decision case by case

Imagine that the new prompt fixes ambiguous case E06 while newly summarizing E07 as “subscription canceled,” despite the customer's instruction to keep the current term open. Both versions could finish with the same total number of passes. E07 is still a regression, and the team should hold the candidate until it resolves that failure. This is an illustrative decision example, not a second measured model run.

Compare every sampled slice as well as the total: category, language, source type, document shape, risk class, or workflow stage when those fields exist in your case register. Count timeouts, refusals, malformed responses, and missing outputs as outcomes; do not remove them from the denominator. For variable outputs, repeat each case under the same conditions and report pass frequency by case. A single success among several attempts is an unstable result, not proof that the change is safe.

A twelve-case fixture can expose a particular defect but cannot establish a population accuracy rate. Report the number of cases, how they were selected, class balance, number of runs, and missing slices beside any summary score. The NIST evaluation paper explains why a score on fixed prompts needs care when generalized beyond those prompts.

There is no useful universal case count. Begin with coverage of each release-blocking failure and the ordinary routes the workflow actually handles; then add cases where a slice still has too little evidence to judge a change. Keep a separately sampled, representative set if you also need an estimate of live performance. A hand-picked regression suite and a production-rate estimate answer different questions, so report them separately.

Diagnose a failure before changing the prompt

Open the input, expected result, raw output, and grader reason. A failure can come from the candidate, an incorrect expectation, changed source data, the output parser, or the grader. In the published fixture, the first grader searched only the evidence field and reported three failures even though the required fragments appeared in summary. After that search boundary was repaired, the retained run passed 12 of 12 cases. Later negative mutations exposed duplicate-ID overwrite, numeric-ID coercion, and an extra field that could hide a forbidden phrase; the current grader rejects those input shapes.

Make the smallest correction that matches the cause. If a label is wrong, inspect the routing rule and adjacent cases. If a protected identifier is missing, decide which output field must contain it. If the grader fails a correct negation, fix or replace the grading rule rather than teaching the model to avoid ordinary language. If reviewers disagree on the expected action, settle the policy first. Rerun the full set after any such change.

Adapt the case design to other AI tasks

The same structure transfers, but the expected result and grader must reflect the task. A document extractor needs field values, types, null handling, and provenance. A retrieval-based answer needs separate checks for finding the right passage, answering from it, and abstaining when the corpus lacks an answer. A tool-using agent needs checks for allowed tools, arguments, authorization, and whether a side effect actually occurred. A support label alone would miss all of those failures.

Keep the cases that represent real work and add a case whenever a reviewed defect escapes. Record why the old set missed it. Retire a case only with a reason and a reviewed replacement or policy change. Once the set is stable, the separate prompt revision guide goes deeper into making and judging one controlled prompt change.