A failed eval does not automatically mean the prompt is wrong. The case may be ambiguous. The grader may reward the wrong behavior. A tool may have returned stale data. The candidate may have been run with a different model setting. This prompt makes that diagnosis happen before anyone “improves” the wording.
Use it only after you have a baseline
This is not a prompt generator. It expects an exact baseline prompt, stable case IDs, per-attempt results, a frozen run configuration, and a grader contract. If you have only a few complaints and a vague memory of the old prompt, first reconstruct the evidence. Otherwise every change will be impossible to interpret.
The minimum input packet is:
- Task contract: the real job, source boundary, allowed actions, required output, and release authority.
- Baseline prompt: exact text with a version or SHA-256, not the latest mutable copy in a dashboard.
- Run configuration: model/version, reasoning or sampling settings, tools, response schema, data snapshot, date, and attempt policy.
- Regression cases: stable IDs, inputs, expected behavior, must-pass rules, and protected legacy behavior.
- Baseline results: one row per case and attempt, including failures, timeouts, refusals, and malformed outputs.
- Grader contract: the exact deterministic checks, rubric, human-review rules, thresholds, and known limitations.
Aggregate accuracy is not enough. A score of 92% could hide a regression on every authorization case while improving several easy formatting cases. The prompt asks for case-level evidence so it can see that difference.
Its most important move is the measurement audit
Before suggesting a new instruction, the prompt classifies each failure:
| Class | What it means | Correct next move |
|---|---|---|
PROMPT_DEFECT | The current wording permits or encourages the observed mistake. | Test a minimal prompt change. |
CASE_DEFECT | The input, expected behavior, or labels are contradictory or incomplete. | Repair and re-review the case. |
GRADER_DEFECT | The evaluator rejects acceptable behavior or rewards an unsafe shortcut. | Fix and calibrate the grader before interpreting scores. |
TOOL_OR_DATA_FAILURE | The model lacked, misread, or received faulty external evidence. | Repair the tool/data path and rerun the baseline. |
RUN_CONFIG_CHANGE | The comparison changed model, tools, schema, or settings. | Restore a matched configuration. |
MODEL_NONDETERMINISM | Repeated matched attempts do not show a stable behavior. | Increase attempts and report pass frequency. |
UNRESOLVED | The available evidence does not identify a cause. | Collect discriminating evidence; do not guess. |
This step protects the prompt from becoming a landfill for every system problem. If a grader requires the literal word “approved” even though the task contract forbids approval language, adding that word to the prompt would improve the score and damage the product.
The candidate is a diff, not a rewrite
Once a genuine prompt defect is supported, the prompt proposes one minimal change set and shows it as a unified diff. Each changed line must name the case it targets, explain why the old wording allowed the failure, and identify a new regression the change might create.
That discipline discourages generic additions such as “be very accurate,” “think carefully,” or “always follow instructions.” Stronger adjectives rarely tell the model which source wins, which status is closed, what to do when evidence conflicts, or which action needs approval. A useful revision changes a decision rule, source priority, boundary, or output schema.
Protect passing behavior explicitly
Teams tend to stare at failed cases and forget the much larger surface that already works. The prompt asks which passing cases depend on the wording being changed. Those become protected legacy behavior in the comparison.
Suppose a support router confuses “billing question” with “refund request.” The fix might clarify that a refund requires an explicit request to reverse a charge. But an overbroad rule—“never choose refund unless the word refund appears”—could break users who say “I want my money back.” A nearby edge case should catch that before release.
Precommit the release decision
The candidate prompt is only a hypothesis. Before results arrive, define the matched comparison and release rules: same cases, model and settings, tool definitions, data snapshot, attempt count, graders, thresholds, and timeout handling. Keep failures in the denominator. Include old passing cases, the cases that motivated the change, adjacent cases that probe overcorrection, and a holdout that was not used to write the candidate.
The exact prompt uses five decisions:
READY_FOR_MATCHED_EVAL— a candidate and test plan exist, but no candidate results were supplied.HOLD_MEASUREMENT_DEFECT— a broken case, grader, data/tool path, or configuration blocks interpretation.HOLD_INSUFFICIENT_EVIDENCE— the baseline cannot be reconstructed.ACCEPT_CANDIDATE— executed matched results satisfy every precommitted rule.REJECT_CANDIDATE— at least one release rule fails.
“The outputs look better” is deliberately absent.
Worked diagnosis: one failure, two very different causes
In the fictional companion fixture, a support-routing prompt has one genuine ambiguity: it does not define how to distinguish a request for billing information from a request to reverse a charge. A case-level failure supports a narrow decision rule and an adjacent paraphrase case.
A second failing row is different. Its grader marks a correctly bounded answer invalid because a required reference answer is missing. The right decision is HOLD_MEASUREMENT_DEFECT, not another paragraph in the prompt. That single distinction is the reason to use this asset instead of asking a model to “rewrite my prompt for better results.”
Use the workbook as the source of truth
The downloadable workbook is a text-first folder rather than a proprietary spreadsheet. It includes a task contract, case registry, run ledger, change record, fictional example, and a Python standard-library gate. The gate verifies stable identities, prompt hashes, matched run configuration, valid statuses, case coverage, and preserved failure rows.
It does not decide whether the grader is right. That remains a separate calibration and review problem. A structurally perfect regression ledger can still measure the wrong behavior.
Read the verification label correctly
The exact prompt was checked against the synthetic packet, and the deterministic gate was exercised with complete records and mutations for missing cases, invalid hashes, configuration drift, and unsupported denominator handling. We did not run an independent model session or a production evaluation. The asset is therefore labeled Structure Checked.
For the complete workflow—from selecting failures through the release record—use How to Improve an AI Prompt With a Regression Set. When your cases are still informal, begin with Build an AI Evaluation Set That Catches Regressions.
Bottom line: do not let the latest failure rewrite the whole system. Audit the measurement, isolate the prompt defect, make one reviewable change, and require matched case-level evidence before the word “improved” appears.
