Choose matching audio and make the reference
Start with the output you intend to publish. A searchable internal transcript, public captions, and an article derived from an interview need different final checks. For a short recording, review all the speech. For a long one, choose clips from the beginning, middle, and end, plus difficult audio, unfamiliar names or numbers, and any silence or music-only interval. State the chosen time ranges and call the result a sample WER; it is not the exact error rate of unreviewed minutes. There is no sample length that fits every recording. If you need to create the source files first, use the local transcription guide. Microsoft’s speech evaluation guidance also calls for human-labeled, representative audio when assessing a model.
Listen to each chosen clip and write the reference without copying the AI output. Note the audio filename and time range beside the text. Replay uncertain words; if you still cannot hear them, ask a second listener or mark the span unresolved. Do not put a guess into the reference and call it ground truth. Decide whether the reference is verbatim, including fillers and false starts, or edited for readability. A reference that silently removes spoken words will make a faithful verbatim transcript look wrong.
Save one reviewed reference and one uncorrected machine transcript as UTF-8 text files. Keep the original AI output even after you make corrections. Put speaker labels and timestamps in a separate review note if the scoring files contain only spoken words. Compare matching audio spans: a missing opening, different audio track, or extra outro can dominate the score even when the shared speech is accurate. If you are comparing two transcription systems, give them the same held-out clips, reference, and text rules.
If no full reference exists, make and review one for the selected clips before calculating WER. You can still listen for critical errors in the rest of the recording, but label that work as human review rather than a measured accuracy percentage. A model confidence number or a fluent-looking transcript is not a substitute for reference text.
Decide how words will be counted
The downloadable Python checker applies Unicode NFKC, case folding, straight/curly apostrophe normalization, and Unicode-aware alphanumeric tokenization. It ignores most punctuation while retaining an apostrophe inside a word. It does not turn “September fifteenth” into “September 15,” or “four thousand eight hundred fifty dollars” into “$4,850.” Record the language, script version, reference version, and those rules beside every score. A different tokenizer can give a different WER for the same spoken content.
| Difference | What this checker does | What the reviewer still checks |
|---|---|---|
| Case and ordinary punctuation | Normalizes case and splits at most punctuation. | Restore punctuation needed for meaning or readable captions. |
| Curly versus straight apostrophe | Treats the two forms alike inside words. | Check whether the actual word or contraction is correct. |
| Written versus spoken numbers | Counts their different tokens as edits. | Listen to the value, unit, date, and currency; do not dismiss a wrong amount as formatting. |
| Names and specialist terms | Counts lexical edits without judging their importance. | Confirm the exact spelling and identity from the audio and an authorized source when needed. |
In our fixed 16-word example, the same spoken words scored 0.0000 with this checker and 0.1250 with JiWER 4.0.0’s default processing because of case and punctuation attached to words. Choose the rule before looking at the score; changing it afterward makes a trend misleading. If you need numbers normalized, define and test that extra step on both texts, and still listen to confirm the actual value.
Run the comparison in a browser or with Python
For a short pair of texts, the browser-local transcript checker needs no installation. Paste the human-checked text into Human-verified reference and the uncorrected output into AI transcript. Both boxes have a 5,000-character limit. Optionally put must-get-right names or terms in Protected terms, one per line, then click Measure accuracy. The tool reports WER, character error rate (CER), reference-word count, and flags to investigate. Download its CSV or JSON report if you need to hand the review to someone else.
We entered the Atlas pair shown below and protected Atlas and Mina Chen. The live tool returned WER 29.63%, CER 35.04%, 27 reference words, and four flags. It flagged the missing Atlas term, a likely-name mismatch, the date, and the amount. Two flags concern the same wrong name; four flags do not mean four separate speech errors. The date and amount were written differently on the two sides, so those flags ask for a listen rather than proving the spoken values were wrong. The browser tool compares text on the current device, but it cannot score an empty transcript and does not show the Python checker’s separate substitution, deletion, and insertion counts.
For repeatable file-based checks, download the Python WER checker, place it beside two UTF-8 text files, and run:
python3 transcript-wer-check.py \
reviewed-reference.txt \
automatic-transcript.txtWER is (substitutions + deletions + insertions) / reference words, often shortened to (S + D + I) / N. The script prints those counts, the two token counts, its normalization name, the number of comparison cells, and WER. For a deliberately altered 16-word fixture, it counted S=4, D=1, I=0: 5/16=0.3125. It prints totals rather than a highlighted word-by-word alignment, so keep the audio nearby when locating the consequential differences.
The browser and Python tools have different token rules and input limits. Our Atlas example produced the same WER in both, but that one agreement does not make the tools interchangeable. Record which tool and reference version produced each score; do not combine their results into one trend.
Read the score beside the words it missed
In our 9.17-second synthetic English test, the reviewed reference was:
“The Atlas review begins on September fifteenth. The budget is four thousand eight hundred fifty dollars. Do not approve the release until Mina Chen checks the source.”
MLX Whisper returned this text on the clean clip:
“The Adless Review begins on September 15, the budget is $4,850. Do not approve the release until Mina Chen checks the source.”
The Python checker counted 27 reference words, 4 substitutions, 4 deletions, and no insertions: WER=8/27=0.2963, or 29.63%. The pink-noise variant said “Ageless” instead of “Adless” and received the same WER. This short synthetic test shows a name error and a number-format difference. It does not tell us how often MLX Whisper makes errors on real interviews.
Some dashboards call 1 − WER “word accuracy.” That would display 70.37% for this example. It is only the complement of this particular edit score: insertions can push WER above 100% and make 1 − WER negative. It also gives the wrong name no more weight than any other edit. Report the actual WER, reference-word count, sample, and text rules rather than presenting “70.37% accurate” as a complete quality verdict.
| What changed | How to judge it | Action before publication |
|---|---|---|
| Atlas became Adless or Ageless. | The name identifies the subject; WER gives it no extra weight. | Replay that moment, verify the name, and correct the published text. |
| “September fifteenth” became “September 15.” | Different lexical tokens may express the same date. | Confirm the date by listening; record the accepted display form separately from lexical WER. |
| The spoken budget became “$4,850.” | Different formatting accounts for several edits, but the value may still be right or wrong. | Check every digit and the currency against the audio. Treat a mismatch in value as a release blocker. |
| “Do not” and Mina Chen were preserved. | That is an observation for this fixture, not proof that other negations or people are safe. | Check them again in each recording being published. |
The test record retains the commands, reference and observed text, and reported outputs. It does not include the source audio, so readers can recalculate the text score but cannot independently re-listen to that model run. For their own material, they should keep the audio and time ranges with the review record.
If a score looks surprisingly high, first check that both texts came from the same audio track, language, and time range. Then inspect number and punctuation rules; a spoken amount written as digits can add several lexical edits without changing its value. If the score is low but a name or negation is wrong, correct that passage before publishing. The score describes a sample; the listening check determines what the error means.
Review meaning, timing, and speakers
Replay each flagged or consequential stretch. Log audio time | reference | AI text | issue | correction | reviewer. For the Atlas example, record Atlas → Adless, the clip time you heard it at, and the confirmed spelling; do not invent a timestamp from the text alone. Save the corrected transcript separately from the raw machine output so the next score can be traced to its input. The publication checklist has fields for status, reviewer, audio time, action, and note. Mark a check not applicable only with a reason, such as caption timing for an audio-only article.
- Names, numbers, and instructions: verify every person, organization, product, amount, unit, date, URL, negation, and ordered step that someone might quote or act on. A single substitution can reverse the meaning while barely moving WER.
- Captions: play the finished captions with the video. Check the first, middle, and final spoken words against the timeline, then inspect line breaks near names, negation, and speaker changes. Plain-text WER does not measure synchronization or caption readability.
- Speaker attribution: compare who said each important phrase with the audio. A transcript can contain the right words under the wrong name. For a scored multi-speaker workflow, use the separate speaker-attribution guide.
- Claims carried into an article: compare each published claim with the recording and any needed primary source. A faithful transcript can still contain a speaker’s mistake or an unsupported assertion; use the source-fact audit for that later writing step.
Handle silence, long recordings, and other languages
For a silent or music-only span, use an empty reference file. The Python checker follows JiWER 4’s empty-reference convention: empty reference plus empty output scores zero; any generated words count as insertions. Our six-second digital-silence fixture returned an empty output and wer=0.0000. Report silence results separately from speech WER so a zero-denominator case is visible. Listen to the span before calling it silence; faint speech or background dialogue changes the reference.
For many clips, keep each clip’s error counts and reference-word count. To report one pooled speech WER, add all substitutions, deletions, and insertions, then divide by the sum of reference words. Do not average clip percentages without accounting for their lengths: two errors in 100 reference words plus one in 10 is 3/110, not the average of 2/100 and 1/10. Break out difficult conditions such as noise or specialist vocabulary so a good overall score does not conceal a failing subset.
Keep a row per clip with audio range | condition | N | S | D | I | WER | consequential errors. That makes it possible to recalculate the pooled score and see whether a poor result came from noise, names, or a mismatched file. The Python checker stops by default above 25 million token comparisons; its memory use is bounded, but running time still grows with the product of the two token counts. For a large corpus, score representative clips or use an optimized tool such as JiWER.
If speech overlaps or a speaker trails off, first decide what belongs in the reference. Do not force two simultaneous voices into one supposedly exact word sequence without recording that choice. If audio and text are in languages without spaces between words, this checker’s alphanumeric runs are not reliable linguistic word boundaries. Use a language-appropriate tokenizer or character error rate (CER), which counts character edits relative to reference characters, and name the metric in your report. A CER from Chinese text and an English WER measure different units. The original Unicode test confirmed that Chinese characters were retained; it did not validate Chinese word segmentation.
Make the release decision for the actual use
Decide what the transcript will be used for before setting a pass rule. No WER threshold approves every use. Record the sample score, then check the passages that could change a reader’s action or understanding.
| Intended use | Additional review | Hold publication when |
|---|---|---|
| Internal searchable draft | Label it as unverified; check names and terms people will search for. | A reader could mistake unresolved text for an approved record. |
| Public transcript or captions | Listen through the published sequence; check timing, names, quantities, negation, and speaker turns. | A material word, speaker, or timing error remains unresolved. |
| Article, minutes, or summary based on the transcript | Verify attributed claims and numbers against the audio before carrying them into new prose. | The source cannot support a consequential statement or attribution. |
| Medical, legal, or other consequential instructions | Use the responsible qualified reviewer and the project’s required approval process; inspect the complete relevant passage. | Any consequential wording is uncertain or the required reviewer has not approved it. |
Before publishing, save the audio name and tested ranges, reference version, tool and text rules, per-clip counts, corrections, reviewer, and decision. For the Atlas fixture, a useful note is hold: Atlas was transcribed as Adless; replay the name and confirm the date and budget against the audio. Once those checks are complete, update the note with the corrected text and who approved it. Keep any unresolved name, value, negation, or attribution out of the public transcript until it can be verified.
