How to Improve an AI Prompt With a Regression Set

This guide begins where the basic eval guide ends

If you do not yet have stable cases and testing criteria, start with our evaluation-set guide. Here, assume you already have a baseline prompt, a small case set, and real results. The work is now change management.

The unit of progress is not a better paragraph of instructions. It is a candidate that passes predefined release rules under a matched comparison.

Step 1: freeze everything that could explain the result

Copy the baseline prompt into version control or a dated record and calculate its hash. Record the model/version, reasoning or sampling settings, tools, tool definitions, response schema, retrieved-data snapshot, attempt count, timeout policy, and grader versions.

This may feel bureaucratic until a team compares a candidate on Tuesday with a baseline from Friday and discovers that the model, search index, and response schema all changed in between. If several variables move at once, a score difference cannot tell you which change mattered.

Give the run configuration a stable ID. Both baseline and candidate rows must reference that same ID. When a configuration change is intentional, treat it as a separate experiment.

Step 2: inspect failures at case level

Never start from the average alone. Open every failed case, its source input, output, grader result, and any tool trace needed to understand the claim. Then classify the failure before editing:

• prompt defect;
• case defect;
• grader defect;
• tool or data failure;
• run-configuration change;
• model nondeterminism;
• unresolved.

Two kinds of evidence are especially revealing. First, look for multiple failures that share one decision error; that cluster may point to a missing rule. Second, look for the same mistake under prompts that differ; if the tool result is wrong in both, rewriting the prompt is unlikely to help.

Step 3: audit the grader as if it were production code

A grader is another specification. It can be wrong, underspecified, brittle, or gamed. For deterministic checks, test positive and negative controls. For a rubric or model grader, build examples of clear pass, clear fail, and difficult disagreement, then compare with qualified human judgment.

Watch for common grader failures:

• requiring an exact phrase when paraphrases are acceptable;
• rewarding verbosity even though the user needs a short answer;
• checking format while ignoring unsupported facts;
• marking a refusal wrong when the requested action exceeds authority;
• using reference answers that leak into candidate generation;
• dropping timeouts or malformed outputs from the denominator.

If the measurement is broken, stop. A prompt can become exceptionally good at satisfying a bad grader.

Step 4: state a causal hypothesis

Do not write “improve classification accuracy.” Write a claim that can be disproved:

Cases RT-07 and RT-12 fail because the baseline prompt never defines when a billing question becomes a refund request. Adding one decision rule based on the user's requested action should fix those cases while preserving RT-01, RT-03, and RT-05.

That sentence tells you what to change, what to test, and what must remain stable. It also reveals the adjacent risk: users may request a refund without using the exact word “refund.” Add a paraphrase case before running the candidate.

Step 5: make the smallest useful diff

Edit one instruction cluster. Keep unrelated wording identical. Prefer an operational rule over emotional emphasis.

- Choose BILLING or REFUND based on the customer's message.
+ Choose REFUND when the customer asks to reverse, return, or recover a charge.
+ Choose BILLING when the customer asks only for an explanation, invoice, receipt,
+ payment-method change, or charge identification. If the requested action is unclear,
+ return NEEDS_CLARIFICATION rather than inferring a refund.

The new text names the decision boundary and fallback. “Be careful to distinguish billing from refunds” does not.

Keep a change record linking every edited line to target case IDs and a possible regression. If a line has no case or contract reason, remove it from this candidate. It can be tested later as its own hypothesis.

Step 6: build the comparison before you see results

A useful matched set has four slices:

Use the same cases, configuration, attempt count, tool/data snapshot, and graders for baseline and candidate. If the system is nondeterministic, run enough attempts to report pass frequency by case rather than choosing the best output.

Step 7: decide with gates, not a blended score

An aggregate score is useful, but some regressions must never be averaged away. Define hard gates for authorization, privacy, unsupported actions, required fields, and must-pass cases. Then add slice and aggregate thresholds.

A simple release policy could be:

1. zero authorization or unsupported-action regressions;
2. all must-pass cases pass every required attempt;
3. target-failure pass rate improves by the predeclared amount;
4. protected legacy cases show no material decline;
5. timeouts, errors, refusals, and malformed output stay in the denominator;
6. human review resolves any rubric disagreement;
7. the holdout meets its threshold.

If any hard gate fails, reject or hold the candidate even if the average rises. Record the exact failed rule. That makes the next revision focused.

Step 8: keep the failed candidate

A rejected prompt contains useful evidence. Retain its hash, diff, results, and decision. Future editors can see that a tempting rule was already tried and why it failed. Deleting failed candidates encourages the same experiments to be repeated without learning.

When a candidate passes, add newly discovered cases to the permanent regression set. A suite should grow from real failures and boundary discoveries, not only from imagined examples written once at the beginning.

A practical folder structure

prompt-change-2026-09-10/
├── task-contract.md
├── prompts/
│   ├── baseline-v1.txt
│   └── candidate-v2.txt
├── cases.csv
├── runs.csv
├── inputs/
├── outputs/
├── graders/
├── change-record.md
└── gate-results.json

The downloadable Prompt Regression Workbook provides these contracts as plain files. Its Python gate needs no third-party packages. It checks identities, hashes, case coverage, matched configuration, allowed run states, and duplicate attempts. Keep in mind what it cannot check: whether the case is representative, the answer key is right, or the grader reflects the user's real need.

Use the revision prompt at the diagnosis boundary

The companion Eval-Driven Prompt Revision Prompt is most useful after baseline results have been retained and before a human edits the production prompt. Give it the task contract, exact prompt, configuration, cases, results, and grader contract. It returns a measurement audit, failure clusters, minimal diff, complete candidate, and matched plan.

Do not supply executed candidate results until the test and release rules are frozen. Otherwise the reviewer can unconsciously move the threshold toward the observed outcome.

Common mistakes that make the loop look better than it is

• Testing only the failed example: proves local repair, not regression safety.
• Changing the prompt and model together: destroys attribution.
• Letting the optimizer see the holdout: turns evaluation into training.
• Removing hard cases after disagreement: inflates the score and erases uncertainty.
• Reporting the best of several attempts: hides instability.
• Adding every exception to one prompt: creates conflicting instructions and makes future diagnosis harder.
• Equating a passing grader with user success: ignores calibration and real-world review.

What the included fixture establishes

The fictional support-routing workbook demonstrates the complete record shape and contains one real prompt-defect pattern plus one invalid grader row. The deterministic gate accepts a matched, complete ledger and rejects known mutations: missing cases, prompt hashes that are not frozen, configuration drift, duplicate attempt identities, and unsupported status values.

No live model comparison was run for this article. The examples therefore teach the method without claiming that the candidate prompt improves a current model. Replace the fixture with your own reviewed cases and run results before making any release decision.

Bottom line: prompt work becomes engineering when you can name the failed case, defend the root cause, show a small diff, reproduce both sides under the same conditions, and reject a prettier candidate when it breaks an important old behavior.