How to Evaluate RAG for Document Question Answering

Define what a correct answer needs to contain

Start with the questions people ask of your documents. Before running a candidate system, label the source needed for each answer, the fact that source supports, and any condition the answer must keep. A return-window answer, for example, needs both the 30-day period and the rule for opened consumables when the question asks about them. Keep the labels separate from model output so the candidate cannot set its own answer key.

The practice fixture has six fictional policy files and 12 questions. Nine have an answer in one file; three ask about facts the files never state. For this exercise, a supported answer gives a short response, the source filename, and a copied supporting sentence. An absent fact returns Not stated in the corpus with null source and evidence fields. These strict fields make mistakes visible. A real answer may need several sources or a sound paraphrase, so plan a separate claim-by-claim review for those cases.

Give each question a failure label as well as an answer key. A wrong document at rank one, a missing exception, a citation to the wrong sentence, and a refusal despite available evidence call for different repairs. For a live corpus, record the document ID, revision, effective date, access group, and source path with each passage. Without those fields, a plausible citation may point to an old policy or a file the user was never allowed to see.

Build the question set from actual reader tasks, including common phrasing and the awkward wording that previously caused a miss. Keep answerable questions, questions whose answer needs an exception, and questions with no answer in the approved corpus as separate groups. A no-answer case should name the missing fact in the answer key; otherwise a reviewer may mistake a topical passage for evidence. Freeze the questions and source snapshot before comparing two versions of your system. Keep some cases out of prompt tuning so you can see whether a later change works beyond the examples you tuned on.

Run the kit and read the retrieval misses

Download the RAG evaluation kit. It contains the corpus in a ready-to-use folder, the 12-question fixture, the Python evaluator, and the prompts, outputs, and reports from the candidate run. A separate older answer artifact is included for comparison. You need only Python's standard library to extract the kit and run the checks:

python3 -m zipfile -e rag-document-qa-evaluation-kit-v1-0.zip .
cd rag-document-qa-evaluation-kit
python3 advanced-ai-replay.py rag
python3 rag-retrieval-eval.py corpus rag-questions.json --top-k 1
python3 rag-retrieval-eval.py corpus rag-questions.json --top-k 3 --trace observed-run/candidate-trace.json

The replay should exit zero. It scores an older completed-answer artifact separately from retrieval; that generator could read all six files. The top-one command should exit nonzero and report 7/9 expected sources in first place. The trace command should also exit nonzero, because the new model output does not match every strict answer and evidence string. Its JSON report still lists every case; a nonzero exit is a prompt to inspect the failed IDs, not a reason to discard the report.

The script ranks whole files using TF-IDF and cosine similarity. At k=3, the expected filename appeared for all nine answerable questions. At k=1, returns.txt was second behind accounts.txt for R01 and R02. With only six tiny files, a top-three hit is an easy bar: half the corpus is in view. Read retrieved_top_k and the individual scores before treating 9/9 as a strong result. The separate threshold field says which ranked results cleared the chosen score cutoff; raising or lowering that cutoff does not change their order.

Try more than one k on your own data. A larger context window may recover a rule that ranked second, as R01 shows, but it can also carry irrelevant or conflicting text into the answer request and use more tokens. If the correct passage remains low in the ranking, inspect the query wording, file extraction, chunking, and reranker before simply increasing k. For each setting, record both source recall and whether the answer kept the rule's conditions; recall alone cannot tell you whether extra context helped.

Follow a retrieval error into the answer

On September 29, the local candidate run used gpt-6-sol through Codex CLI 0.158.0-alpha.2.1. Each of the 12 questions was sent in a separate read-only session with only that question and its top-three retrieved passages in the prompt. The retained event summaries show no tool calls. This is a test of the small fictional corpus, not a measurement of a production RAG application. The kit keeps each prompt and JSON response so you can check what the model saw.

Across the 12-question top-three run, the expected source appeared in 9/9 answerable rankings. The exact fixture checker passed 7/12 completed answers. That number is deliberately strict: it demands the canonical wording and one prewritten evidence span. Reviewing the five failures against the source text found four wording or span differences that kept the supported fact, plus the R05 omission that changed the shipping rule. For R03, the model's “Within two business days” follows the source wording more closely than the canonical phrase “after approval,” whose starting point is not explicit in the source sentence. The observed-run/editorial-review.json file records each judgment. This editorial review happened after seeing the failures; it is not a blinded accuracy study.

Capture the context from your own application

To try the same method on your system, add logging where the answer request is built. Save the question ID and ordered retrieved chunks after permission filters and reranking. Then save the exact chunks inserted into the model request as context. These may differ: a prompt builder can drop a chunk, shorten it, or add text from elsewhere. Copying passages from the index after the response was generated would hide that difference. Finally save the answer, source ID, quoted evidence, and abstention flag. Keep the prompt, model, index revision, filter settings, and run date beside the trace for debugging.

Open observed-run/candidate-trace.json in the kit for a complete 12-record example. Every record has id, ranked retrieved passages, context passages that went into the prompt, answer, source, evidence, and abstained. The evaluator accepts an empty retrieval and context array for an unanswerable question when the retriever found nothing. For a private corpus, keep sensitive passages in your own environment and share only redacted review records.

Export records for all question IDs and run python3 rag-retrieval-eval.py corpus rag-questions.json --top-k 3 --trace your-trace.json, using the exact source snapshot that produced the trace. The checker confirms that each logged passage belongs to its named source file, context came from the first k retrieved passages, and a passing answer's quoted evidence was in the context. trace_retrieval_recall_at_k scores the recorded ranking. The separate retrieval_recall_at_k field is the script's own TF-IDF baseline over the local files. Compare those only when both used the same corpus and questions.

Check claims and refusals after the exact score

A strict string check is useful for catching input errors and regressions. It cannot decide whether a paraphrase is faithful. Read each failed answer beside the passage the model actually saw. For the shipping question, the file says standard shipping is free at 75 US dollars or more before tax and that oversize items are excluded. “Orders of 75 US dollars or more before tax” drops the exclusion. The right filename and a valid quoted sentence do not rescue that broad answer.

Review each factual claim with its citation. Match the subject, condition, time frame, and scope. When an answer combines two files, attach each claim to the passage that supports it and check for missing citations. AWS's RAG evaluation documentation treats retrieval quality, answer faithfulness, and citation coverage as different measurements; the single-source exact checker in this kit does not collapse them into one semantic score.

Check refusals in both directions. On R10, the absence of a warranty period makes refusal correct even though a shipping file was retrieved. On R01 at k=1, refusal signals a retrieval problem because the return rule exists but never reached the model. Report correct refusals on no-answer questions and wrongful refusals on answerable questions separately. A similarity score alone cannot decide whether the retrieved passage states the requested fact.

In your review sheet, track whether the required source was retrieved and actually sent, whether the answer and its conditions are supported, whether the citation is right, and whether any refusal is appropriate. Mark each question before calculating totals. That makes the denominator visible: 9/9 here measures source presence for the nine answerable questions, while 7/12 measures strict answer-and-evidence matching across all twelve. Neither number measures access control or the cost of serving a user request.

Add the cases your documents can actually fail

The kit does not contain long documents, revisions, tenant permissions, or injected instructions. Build those tests from your own system before using a score to approve a release. Write the expected behavior before running the candidate, and keep the trace that shows whether it happened.

These are test designs, not results from the included run. The downloadable checker expects one source and one canonical answer per question; it cannot automatically grade a two-source claim, a tenant leak, or a conflict of authority. Give those cases their own review rules. A tenant leak should stop a release even if the average answer score looks good.

Repair the stage that failed

If the needed passage is missing, check the index snapshot, access filter, document version, chunk boundary, and rank before changing the answer prompt. The R01 top-one run shows why: the model could not quote a return rule it never received. If the passage is present but the answer drops an exception, inspect the prompt and add a claim-level check for qualifiers. If the answer is sound but the exact checker rejects different wording, update the accepted-answer contract or send the case to human review; do not count that as a retrieval miss.

Save the failed query and trace, change one component, then rerun the development cases and an untouched holdout set. Report failures by type beside latency and cost when those matter to your application. The included files are about 152 words of synthetic English policy text and the baseline tokenizer recognizes ASCII letters and digits. Use a retriever and answer review appropriate to your corpus language and length. The local numbers show how to find and explain failures; they do not estimate production accuracy.