What it is, and why we built it
I’m Donna. The Record Test Bench is a public leaderboard that tests generative AI on legal transcript editing, scored against certified transcripts.
A lot of court reporters and legal transcriptionists worry about losing their jobs to AI. We didn’t want to guess how close the models are, so we set out to measure it. Here are the results.
The results
These are the audio full-chain scores. Each model works from the deposition audio: it proofreads, corrects speakers, then formats.
- Legal-structure F1: 53.7%. gemini-3.8-flash (low) is highest, tied with gemini-4-argon (low) at 48.1%.
- Speaker-name accuracy: 63.9%. gemini-3.8-flash (low) again.
- Best verbatim word error rate: 20.2%. gemini-3.1-flash-lite (minimal). Lower is better, and 20.2% means substituted, missing or added words add up to about one for every five reference words.
Legal structure is the part that makes a transcript a legal transcript: testimony openings, proceedings closings, examination headings, attorney bylines and parentheticals. Speaker-name accuracy asks whether each turn is credited to the right person.
If you review drafts from audio, those are the three places to spend your attention: the words, who said them, and the headings and bylines that frame the testimony.
The text-only track starts from the dataset transcript instead of audio. Its best legal-structure F1 is 25.8% (claude-sonnet-5-5) and its best speaker-name accuracy is 52.5% (gpt-6.1-sol). The two tracks are scored separately. For a closer look at speaker-name accuracy, read Gemini, Sol and Sonnet: speaker tracking is still not fixed.
The leaderboard’s headline table: audio full-chain models on the left, text-only models on the right.
How it’s scored
The bench uses five public-record depositions, 20.8 hours of English audio, and reference transcripts prepared from certified transcripts.
Each model is scored on three things: the words, the speakers and the legal structure. In Will AI replace court reporters? I said we were building a public test and would post the numbers when it ran. It ran, and they are posted. For what to check in an AI edit before you accept it, see ASR, generative AI and reviewing an edit. For the difference between verbatim and normalized word error rates, read Words before punctuation.
We built the bench and published it in the open, so you can check every number.
See every score
- The leaderboard, with every model, metric and per-file download
- The methodology and metric definitions
- The public code for the scorers and runner
- The Hugging Face datasets: the corpus and outputs and the scores
The paper is coming soon on arXiv.
Conclusion
LLMs are far behind humans on this bench. The best audio-track legal-structure F1 is 53.7%, the best speaker-name accuracy is 63.9%, and the best verbatim word error rate is 20.2%. AI is nowhere near replacing legal transcribers yet.
For reporters and transcriptionists, that means AI can assist with a draft, but certified human work is still essential. Someone has to catch the wrong words, the wrong speaker and the missing headings before a transcript is certified.
We’ll re-run the bench as new models are released and publish the results on the leaderboard. Follow it to see how the numbers move.
Watch this space.