How to Analyze Customer Feedback With AI Using a Calibration Set

Choose the decision before choosing labels

A support queue needs to know which team should see a message and which mistakes require immediate review. A product team may want to know which obstacles recur. A researcher exploring unfamiliar feedback may need to discover themes before fixing a taxonomy. These jobs call for different outputs. A single label can be useful for routing, while one message may need several theme tags when you count customer problems.

Write down the unit you will count: messages, unique customers, orders, or product sessions. If one customer sends five follow-ups about the same failed upload, five messages are not necessarily five separate problems. Keep the original text, stable ID, channel, product, and time period alongside any AI label so a reviewer can trace a finding back to its source. Decide who will act on each category before processing a large batch.

Write a taxonomy a reviewer can apply

The downloadable exercise uses six mutually exclusive labels. Start with the rules below, then ask reviewers to apply them to your own feedback. The recorded 30-row run did not test every boundary in this table.

For this one-label exercise, classify the issue the customer most clearly asks someone to resolve. In a real queue, a message with two independent requests needs two issue records linked to the same source message, or a reviewer who can split and route them. If neither the text nor its permitted context supports a label, put the message in a needs_review queue. The supplied scorer accepts only the six exercise labels. It cannot score two issues from one message or an abstention, so evaluate those separately before using this policy in production.

Build two samples for two different questions

Use a calibration or challenge sample to expose label boundaries. Deliberately include near misses, long comments, mixed issues, short fragments, and the channels you plan to process. Keep a separate random or otherwise representative sample if you need to estimate prevalence. A challenge set that contains five examples of every class is useful for exercising the scorer, but its class proportions were chosen by the editor. They are not customer proportions.

Have two reviewers label some rows independently before they see model output. For every disagreement, record the text, both proposed labels, the final decision, and the rule that settled it. If the reviewers cannot agree after reading the rule, revise the rule or keep the row in review. Do not quietly turn a disputed label into ground truth. Freeze the reference labels before comparing a new prompt or model; otherwise you may end up changing the answer key to match the prediction.

The 30-row calibration CSV has five synthetic examples per class. Every message is short and fairly direct. It is enough to reproduce the scoring path, but it does not test reviewer disagreement, mixed intent, sarcasm, other languages, very long feedback, or the frequency of issues in a real customer population. Those cases need their own labeled review set.

Run the six-label exercise without revealing the answers

Download the scorer, the calibration CSV, and the retained model output into one folder. The CSV has id, text, and expected_label. Give the model only id and text; keep expected_label for evaluation. From that folder, this standard-library command makes an input file without the answer column:

python3 - <<'PY'
import csv, json
from pathlib import Path

with open('customer-feedback-calibration.csv', newline='', encoding='utf-8') as file:
    rows = [{'id': row['id'], 'text': row['text']} for row in csv.DictReader(file)]
Path('feedback-input.json').write_text(
    json.dumps({'records': rows}, ensure_ascii=False, indent=2) + '\n', encoding='utf-8'
)
PY

In an approved chat tool, attach feedback-input.json, paste the policy below, and save the complete JSON response. In an API workflow, send the same records and instructions, then check that every input ID appears once in the response. If the tool truncates a large batch, split it by ID and check for missing or duplicate IDs after combining the outputs. The downloadable model output came from an earlier run; this exact prompt wording has not been tested across models.

Classify each record independently. Return one JSON object with a "records" array.
For every input ID, return exactly: id, label, summary, evidence.
Use one label only: bug, feature_request, praise, access, billing, usability.

bug: an existing function fails or returns the wrong result.
feature_request: a new capability is requested.
praise: positive feedback with no unresolved request in the message.
access: sign-in, invitation, permission, or entitlement problem.
billing: charge, renewal, receipt, tax, seat count, or invoice issue.
usability: a working function is difficult to find, understand, or use.

Choose the clearest issue the customer asks to resolve. Do not infer a cause,
severity, customer identity, or resolution. Write a short factual summary.
Copy the smallest exact phrase from that record's text that supports the label.
Preserve each ID. Return each ID once and do not add fields or commentary.

For example, the retained record for F01 has label: "bug", summary: "CSV export button remains disabled", and evidence: "stays disabled". Check the evidence against the original message before trusting the summary. The scorer verifies that the phrase occurs in the same source row; it cannot tell whether that phrase actually justifies the chosen label or whether the summary misses an important detail.

Save the AI response as my-feedback-output.json. It must have the same top-level {"records": [...]} shape as the retained output. Then run:

python3 customer-feedback-calibration-score.py customer-feedback-calibration.csv my-feedback-output.json

Start with valid_input. A missing or duplicate ID, empty summary, unknown label, or evidence phrase absent from its own source row invalidates the run before you interpret any metric. The command exits nonzero for invalid input. It checks exact text containment, so a phrase that happens to occur in two rows can still pass; a reviewer must judge whether the phrase supports the label.

Once the structure passes, inspect the confusion matrix and each class's precision, recall, F1, and support. The classification-report definitions are useful if those terms are new: precision concerns predicted members of a class, recall concerns actual members, and support counts reference examples.

Inspect one wrong classification

The retained output correctly labels all 30 clean rows and returns macro F1 1.0. Run python3 advanced-ai-replay.py feedback with the replay script in the same folder to reproduce that threshold check and three malformed-output rejections. A perfect score here shows that the synthetic fixture, output contract, and scorer agree; it does not establish accuracy on live feedback.

To see a substantive classification error, make a copy of the retained JSON and change only F13 from bug to usability. The source says the mobile page freezes during an upload. That is a failed function under this taxonomy, even though a user may also find the experience frustrating. Score the changed copy with the ordinary scorer command. In this one-error example, the input remains valid, overall accuracy becomes 0.9667, macro F1 becomes 0.9663, bug recall falls to 0.8, and usability precision falls to 0.8333. The confusion matrix places one actual bug under predicted usability.

The next action depends on the consequence. If bugs and usability reports go to different queues, review similar upload messages and add a boundary example to the taxonomy. If the two labels have the same owner, the error may matter less for routing but still distort a report about product defects. Do not choose a release threshold from the clean fixture's perfect score. Set any hard stop for a costly confusion pair using the team's actual decision and a labeled set that contains that kind of case.

Keep difficult feedback visible

Use these examples to review your policy. They were not part of the downloadable 30-row run and have no measured model results.

When a boundary case changes the policy, update the written rule and add examples from both sides of the boundary. Version the taxonomy, prompt, reference labels, and model settings together. Retest the old cases as well as the new ones. A change that fixes access-versus-billing messages can create a new mistake elsewhere.

For the double-charge and sign-in example, a reviewer could create message-17/billing and message-17/access, each pointing to the same source message, then send them to different owners. Count that as one customer message and two issues. Keep vague messages in the review queue until a person can request or find permitted context. To evaluate this workflow, check whether each issue was found and routed; the six-class, one-label confusion matrix cannot measure a missing second issue.

Turn labels into a reviewable business finding

After classification, give the decision owner a table that includes the theme, the counting unit and denominator, the time window, the channels included, a few checked source IDs, the reviewer decision, and the proposed action. Keep a separate queue of uncertain or mixed messages. An executive chart with only label percentages hides the sample design and makes a clean classifier look more certain than it is.

Here is an illustrative reporting calculation, not a result from the downloadable exercise or a claim about any company's customers. Suppose a month has 600 chat issue threads and 400 email issue threads after grouping follow-ups about the same incident. Draw 60 chat threads and 80 email threads at random within their channels. Have reviewers check the model's labels against the original messages. They confirm access problems in 6 chat threads and 24 email threads.

Email was sampled more heavily, so the unweighted 30 / 140 = 21.4% is not the combined monthly estimate. Weight each channel by its share of the 1,000 issue threads: (600 / 1,000 × 10%) + (400 / 1,000 × 30%) = 18%. Report this as an estimate for issue threads in those two support channels during that month, not as a fact about all customers. Only 140 of 1,000 threads were reviewed, so a repeat sample could give a different rate.

Give the access owner the reviewed source IDs and the distinct symptoms to investigate. A label count alone cannot identify the cause of a sign-in problem.

For routing, inspect every high-cost misclassification in a labeled evaluation set and sample other predictions to look for overlooked error patterns. In production, the true label is unknown until a person checks it. Route ambiguous or high-consequence messages to reviewers by rule, and audit a sample across predicted classes so misses outside the flagged class can surface.

For prevalence, define the population and counting unit first, then draw a fresh sample from the period you want to describe. Record exclusions and channel mix. The balanced exercise's five-per-class distribution is not a production trend; a change in channel volume can also change the overall mix.

A bug label records what a customer reported, not why it happened. "Upload freezes" is a symptom; it does not prove the cause of the failure. Keep severity, ownership, and resolution in separate fields with their own evidence. For a proposed feature, preserve what the customer wanted to accomplish as well as the requested feature name. That gives the product team something more useful than a tally of feature requests.

Review new feedback and changes over time

A model's self-reported confidence is not automatically a probability that its label is correct. If you plan to route by a score threshold, first compare scores with human-reviewed outcomes on a relevant holdout and inspect error rates within score bands. Until then, send ambiguous, mixed, and high-consequence messages to reviewers by rule rather than treating a confident number as proof.

Freeze a reference set and rerun it when the prompt, model, taxonomy, or input format changes. Also sample current production messages for human review on a schedule the team can sustain. A stable holdout detects some regressions; fresh reviews can reveal new language, products, channels, and issue types the holdout never contained. When a predicted class rises, check channel volume, duplicate customers, and reviewer-confirmed examples before calling it a product trend.

For your first pass, save the taxonomy, answer-hidden input, model output, scorer report, and reviewer notes together. If the scorer says valid_input: false, repair the record contract. If a consequential confusion appears, revise the rule and retest its boundary. Use a separate representative sample before reporting which problems are most common.