How to Evaluate Multi-Speaker Transcription and Speaker Attribution

Make a reference that matches the recording

Keep the original audio and the system output exactly as delivered. Make a separate reference transcript by listening to the audio, correcting words, and marking speaker turns. Pay particular attention to short replies, names, numbers, and negation. If two voices overlap, mark both in the reference. Give each person a stable anonymous label such as PERSON_A; attaching a real name calls for separate confirmation.

For interval scoring, save reference and predicted turns as RTTM. Each SPEAKER row contains the recording ID, channel, start time, duration, and speaker label. In this row, PERSON_A speaks from 1.20 to 2.40 seconds:

SPEAKER meeting01 1 1.20 1.20 <NA> <NA> PERSON_A <NA> <NA>

The recording ID and channel must agree across the two RTTM files. The clocks must share the same zero point. A ten-second offset from a video edit can make an otherwise good prediction look disastrous. Check one turn near the start and one near the end against the audio, and confirm that no row runs past the file duration. If a tool returns JSON, turn each predicted speaker interval into an RTTM row using its start time, duration in seconds, and original cluster label. Do not stretch a turn across a silent gap to make the rows continuous.

Silence was the decisive correction in this page's example. The supplied WAV lasts 24 seconds, with six voice clips placed in four-second slots. Its PCM samples contain 7.553 seconds of exact-zero trailing audio across those slots. An earlier reference labeled every slot as speech, so its 24-second denominator could not reveal false alarms in silence. The revised reference ends each turn at the synthetic voice signal, rounded outward to 10 ms. It labels 16.48 seconds as speech and leaves 7.52 seconds unlabeled. These boundaries are adequate for a controlled scoring exercise. For a real recording, listen to the boundaries: a nonzero sample could be breath, noise, or music rather than speech.

If you make an analysis copy with FFmpeg, retain the original and note the conversion. Check separate microphone channels before downmixing; combining them can remove useful speaker evidence.

ffmpeg -i meeting-input.aiff -ac 1 -ar 16000 -c:a pcm_s16le meeting-normalized.wav

Run the sample: words, intervals, and attributed words

Download the evaluation kit and extract it into one directory. It includes the 24-second WAV, reviewed and observed transcripts, reference and deliberately flawed RTTM files, speaker-attributed STM files, three scoring scripts, and the commands below. The WAV contains three synthetic English voices. The observed transcript is retained from a local MLX Whisper run; the speaker hypotheses were written by hand to test the metrics. They are not predictions from a diarization model.

python3 transcript-wer-check.py \
  diarization-reference-transcript.txt diarization-observed-transcript.txt

python3 diarization-attribution-score.py \
  diarization-reference-rttm.txt diarization-hypothesis-rttm.txt --step 0.01

The text check returns wer=0.0217: one substitution, “the” for “a,” among 46 reference words. The small RTTM scorer returns 0.45 seconds of missed speech, 0.35 seconds of false alarm during reference silence, and 2.25 seconds of speaker confusion. After mapping the arbitrary hypothesis labels to the three reference speakers, its error rate is (0.45 + 0.35 + 2.25) / 16.48 = 0.1851, or 18.51%. Most of that error comes from the fourth turn, which the hand-written hypothesis assigns to the wrong speaker. The hypothesis also starts one turn early into silence and clips parts of other turns.

Run python3 advanced-ai-replay.py diarization if you want to check the fixture's retained values, audio duration, reference boundaries, and invalid-input exits. This confirms that the files and scorer agree; it does not measure a diarization model. The sample's WER and speaker scores answer different questions and have different denominators.

Next, score the words under their speaker labels. The two included STM files contain one timed utterance per line, in the form recording channel speaker start end text. Install MeetEval in a local Python environment:

python3 -m venv .venv
.venv/bin/python -m pip install meeteval==0.4.3 simplejson
.venv/bin/meeteval-wer cpwer \
  -r diarization-reference-attributed.stm \
  -h diarization-hypothesis-attributed.stm
.venv/bin/meeteval-wer tcpwer \
  -r diarization-reference-attributed.stm \
  -h diarization-hypothesis-attributed.stm --collar 3

On Windows, use .venv\Scripts\python and .venv\Scripts\meeteval-wer. With MeetEval 0.4.3, both commands return 32.61% on the included files: 15 errors over 46 reference words, comprising seven insertions, seven deletions, and the one ASR substitution. The wrongly attributed fourth utterance is seven words long. After MeetEval chooses the best permutation of anonymous speaker labels, those words are missing from one speaker's text and extra in another's. cpWER therefore exposes a delivered-transcript problem that ordinary WER, at 2.17%, conceals. The three-second tcpWER collar is a setting for this short utterance-timed example, not a universal tolerance.

These two WER numbers were produced with different tools. Tokenization and text normalization affect word counts, so use the same policy before comparing systems on your own data. The bundled plain-text checker uses Unicode NFKC, case folding, alphanumeric tokens, and internal apostrophes. If punctuation, contractions, or multilingual text matter, write down the rule you actually used.

Apply the three checks to your own transcript

Start with the files you have. If your service exports only a plain text transcript, you can check WER after making a reviewed reference, but you cannot measure speaker timing or word attribution from that text alone. Ask for timestamped speaker segments or the service's raw JSON. If it exports speaker-labeled segments, save their time spans and text before anyone corrects them. Use the same recording ID, channel, time origin, and duration in every format.

For a single recording without overlap, replace the four transcript and RTTM filenames in the first commands. The bundled RTTM scorer samples time every 0.01 second and finds the best one-to-one mapping between anonymous hypothesis labels and reference labels. It rejects overlapping turns, mixed recording IDs or channels, nonpositive durations, and an empty reference. Run it on one recording at a time. Read its miss, false alarm, and confusion seconds before relying on the combined percentage. A system label called SPEAKER_02 can legitimately map to reference PERSON_A.

To check the delivered words by speaker, make two STM files. Each line needs a recording ID, channel, speaker label, start, end, and utterance text. Keep separate lines at speaker changes. For example, if A says “approve the invoice” from 5.0 to 6.2 seconds and B says “hold the payment” from 6.3 to 7.5 seconds, use two lines. A single sentence timestamp spanning both turns cannot establish which person said each word. Where a word straddles a handoff, listen and decide whether the segment should be split; preserve the original prediction for audit.

Use cpWER when you need to know whether each speaker's final text is right after an optimal label permutation. Use tcpWER when time alignment matters as well; report its collar, because a tighter setting may reject otherwise matching words. RTTM contains no words, so it cannot feed either metric on its own. If you have word-level timestamps, retain them for inspecting short replies and overlaps even if your scorer accepts utterance-level STM.

Use overlap-aware DER when turns coincide

The bundled standard-library scorer deliberately stops when the reference or hypothesis has overlapping speaker intervals. Keep both speakers in a real overlap and use a scorer that accounts for simultaneous speech. The kit includes a pyannote.metrics wrapper for one recording. Give it the audio duration so the evaluation region includes silence as well as speech:

.venv/bin/python -m pip install pyannote.metrics==3.2.1
.venv/bin/python diarization-standard-der.py \
  diarization-reference-rttm.txt diarization-hypothesis-rttm.txt \
  --duration 24 --collar 0

.venv/bin/python diarization-standard-der.py \
  diarization-overlap-reference.rttm diarization-overlap-hypothesis.rttm \
  --duration 3.5 --collar 0

In the first run, pyannote.metrics 3.2.1 returns DER 0.185073, with 0.45 seconds missed, 0.35 seconds false alarm, and 2.25 seconds confused. It closely matches the 0.01-second sampled scorer because this fixture has no overlap and uses zero collar. In the second, separate illustration, two reference speakers overlap from 1.0 to 2.0 seconds, and the hypothesis starts the second speaker at 1.5 seconds. DER is 0.125 with overlap scored. Add --skip-overlap and the same files return 0.0, because the remaining evaluated portion contains no error. The changed number describes a changed protocol, not an improved transcript.

A collar excludes time near reference boundaries; an evaluation region (UEM) says which part of the recording counts. Set both, plus the overlap policy, before comparing systems. For meetings with long pauses, include confirmed silence in the region if false alarms there matter to your product. For a benchmark, follow that benchmark's specified scoring rules rather than substituting the settings in this demonstration. Report the scorer and version, collar, overlap policy, region, and reference speech duration with every DER.

Find the cause behind a bad score

A short “yes” at a handoff can be a deleted word, missed speech, or a correctly recognized word assigned to the previous speaker. Look at WER, timed speaker errors, and the delivered utterance together. For remote meetings, echo may duplicate a voice and separate channels may have different delays. For interviews, a guest who joins late may receive a new cluster only after several words. Those are reasons to inspect the raw timeline before merging labels or changing a threshold.

Report a result that supports a decision

For more than one recording, keep each file's word counts and diarization error seconds. Sum the counts or durations before calculating an overall rate; a five-second clip should not weigh as much as an hour-long meeting merely because both are one row. Include the number of files, reference speech time, speaker counts, and at least one timestamped example of a consequential error. If the delivered transcript displays real participant names, record who confirmed each cluster-to-person link.

Decide acceptable performance from the use case and review capacity. Meeting notes may tolerate a brief speaker mix-up that a legal interview or clinical record cannot. A captioning workflow may care most about missing short replies and overlapping speech. State which errors a human will correct, which files were sampled, and which conditions remain untested.

For the included teaching file, the auditable statement is: one word substitution in 46 reference words; 0.45 seconds missed, 0.35 seconds falsely marked as speech, and 2.25 seconds assigned to the wrong speaker over 16.48 seconds of reference speech; cpWER 15/46 with a deliberately wrong seven-word turn. It establishes that the supplied scoring paths expose different errors. To decide whether your transcription system is usable, replace the hand-written speaker hypotheses with its uncorrected output and score recordings that resemble the ones you will actually process.