17%. That was the legal-term recall of an open-weights speech model the first time we scored it on our own deposition audio. AssemblyAI, the speech-to-text service HeyDonna uses today, scored 73% on the same files. That was its Universal-3.5 Pro model.
A gap like that looks like a model that can’t hear legal testimony. It mostly wasn’t. Most of it was how the model wrote numbers down.
What we tested
We ran NVIDIA Nemotron 3.5 Transcribe 30B-A4B (nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B on Hugging Face), an open-weights model anyone can download, against AssemblyAI. We also scored WhisperX large-v3, an open-source pipeline built on OpenAI’s Whisper large-v3 model. We didn’t call AssemblyAI again for this. We scored the AssemblyAI output already stored for these files. All nine used AssemblyAI Universal-3.5 Pro (universal-3-5-pro). Four came from HeyDonna’s own requests: English, speaker labels on, and a boost list of names and terms from the job details. The other five came from Scribie, a partner service that also uses AssemblyAI’s latest model.
The WhisperX large-v3 output also came from storage: Scribie’s production WhisperX service had already transcribed the same nine files, with its own speaker labels. No new WhisperX calls were made.
To label who is speaking with Nemotron, we tried two ways of matching speakers across chunks of audio. One of them uses TitaNet, NVIDIA’s open speaker-recognition model.
The audio was our internal bench: 9 consented deposition files, 15.84 hours in total. The reference for each file is the delivered transcript, prepared by a human. This is a small internal bench, not a public leaderboard, and I name no case or speaker from it.
We scored three things:
- Word error rate (WER): substituted, missing and extra words, divided by the words in the reference. Lower is better.
- Speaker-attributed WER (cpWER): the same count, but a word only counts as right if it is credited to the right speaker. Lower is better.
- Legal-term recall: of the exhibit numbers, dates, dollar amounts, titled names and other numbers in the reference, how many the transcript got exactly right. Higher is better.
The 17% that wasn’t
Legal-term matching is literal. The reporter’s transcript has “2024”. The open model wrote “twenty twenty four”. You and I hear the same year. The scorer saw a miss.
The same thing happened with dates, times and amounts. “September twenty fourth” against “September 24”. The model often wrote numbers out as words, and the reporter writes digits.
So we added one rule and applied it to both the reference and every model’s output, in exactly the same way: spoken numbers become digits. Then we scored again.
Legal-term recall (higher is better)
The open model’s recall went from 17% to 84%. AssemblyAI’s went up too, from 73% to 88%, and WhisperX large-v3’s from 65% to 86%, because the reference and the output didn’t always write numbers the same way there either.
The words were mostly there. The formatting was different. That is the difference between formatting and hearing.
What was left after fair scoring
Then we made one more adjustment, again identical for the reference and every model. We removed the fillers “uh” and “um”, and stand-alone fragments like “okay,” “yeah” or a repeated “I,” which a clean reporter transcript usually leaves out. Affirmative answers like “uh-huh” stayed in.
So the final scores use one method for everyone: numbers written as digits, uh and um removed, applied the same way to the model output and to the reporter’s transcript.
Here is the full picture, pooled across all 9 files. First, plain word errors:
Word error rate on legal audio (lower is better)
Then the errors once speakers are counted:
Speaker-attributed error, cpWER (lower is better)
Fair scoring narrowed the gap. It didn’t close it.
- Words: AssemblyAI had the fewest word errors at 4.69%. WhisperX large-v3 came second at 6.88%, and Nemotron third at 8.23%.
- Speakers: with each model’s own speaker labels, AssemblyAI’s speaker-attributed error was 8.38%. Nemotron’s was 18.67% and WhisperX large-v3’s 23.28%. WhisperX gets more words right than Nemotron, but gives more of them to the wrong speaker.
- Speaker matching helps: with TitaNet matching speakers, the open model’s raw cpWER was 10.6% across 7 files. We could not rescore that run after normalising, so the chart shows its raw score only, labelled with its 7 files.
- Names: AssemblyAI found 80% of the names, WhisperX large-v3 also 80%, and Nemotron 76%.
Most of what remains is extra words the open model keeps, such as fillers and backchannels the reporter left out, plus some words it heard wrong.
The part that is genuinely good
The open model was fast and cheap. On one NVIDIA H100 GPU it transcribed the 15.84 hours in 516 seconds of processing time. That is about 110 times real time, ranging from 89 to 141 times across the nine files.
Transcription plus speaker matching used about 12 minutes of GPU time, roughly 77 cents at the $3.95-an-hour H100 rate we budgeted with. Our whole Modal bill for the bench came to $5.02. That includes loading the model, setup and a few failed starts.
How we measured: one run on 28–29 October 2026, a single H100 on Modal, with the model served by vLLM 0.32.0. Real-time factor is audio length divided by the time the model spent on each chunk of up to 40 minutes. It leaves out the one-time model load of about 8½ minutes and the network round trip. Counting the round trip from our runner, it was about 75 times real time. GPU cost is measured GPU seconds times $0.001097 a second. The $5.02 is the Modal billing report for the bench app. The AssemblyAI side made no new calls. It reused stored AssemblyAI Universal-3.5 Pro (universal-3-5-pro) output for all 9 files: on 4 files, HeyDonna's production request (English, speaker labels, job-detail word boost), and on 5 files, Scribie's run, which also uses AssemblyAI's latest model.
That is worth knowing. It isn’t enough on its own. Fast and cheap matter after accurate, not before. So HeyDonna keeps AssemblyAI in production.
Why this matters if you work from audio
You will see accuracy numbers in vendor pitches, reviews and forum threads. Before you trust one, ask what it counted.
- Were numbers normalised? If one transcript writes “2024” and the other writes “twenty twenty-four”, a strict scorer calls that an error. Your style guide may not.
- Were fillers counted? A verbatim transcript and a clean one disagree about “uh” on purpose. Either choice is fine, but both sides of a comparison should make the same one.
- Were speakers counted? A transcript can get every word right and still give the answer to the attorney. Plain WER won’t show that. Speaker-attributed scores will.
- How much audio, and whose? Nine files is enough to show a clear gap like this one. It is not enough to rank every model on every kind of proceeding.
None of this changes your job. A draft is a draft, and the person who listens to the audio, fixes the words and certifies the record is still you. For a closer look at checking words against the recording, read Words before punctuation. For who said what, read speaker tracking is still not fixed. For how we score editing models in public, see the Record Test Bench.
What scoring rule would you want applied before you trust an accuracy number?