What dataset and ground truth does this benchmark use?

LibriSpeech test-clean, a public-domain corpus (CC BY 4.0) of LibriVox audiobook recordings with professionally aligned reference transcripts.

Is this benchmark data or a marketing claim?

Every number on this page traces to a public dataset, a scoring script, and raw model output published alongside it — including a downloadable CSV of per-utterance results.

Has VideoText itself been benchmarked here?

Not yet. This pilot tested an open-source Whisper baseline only; VideoText, Deepgram, AssemblyAI, and other tools are planned additions once API access is available.

← Back to home

Research · Phase 1 pilot · Updated 2026

AI Transcription Accuracy Benchmark

Q: How accurate is AI transcription?

In our Phase 1 pilot, an open-source Whisper baseline scored 2.52% word error rate on 25 clean, single-speaker LibriSpeech utterances — a narrow, reproducible data point, not a general accuracy claim.

An open, reproducible study measuring transcription accuracy across tools using identical audio, ground-truth transcripts, and a public scoring methodology. This page publishes our dataset, our code, our results — including where we underperform — not a marketing comparison.

Direct answer

How accurate is AI transcription today?

On clean, single-speaker speech, an open-source Whisper baseline scored 2.52% word error rate in our Phase 1 pilot.

This is one data point from a small, reproducible pilot — not a finished 8-tool study. We're publishing it now, with full methodology and the underlying data, and will expand it condition-by-condition and tool-by-tool as described below. Treat every number on this page as scoped to exactly the condition and tool it's labeled with.

Dataset: LibriSpeech test-clean (public domain, professionally aligned reference transcripts)
Condition tested so far: Clean, single-speaker, read speech
Sample: 25 utterances, 5 speakers, 4.3 minutes of audio

What this page is — and isn't — right now

This is a Phase 1 pilot, not the full "8 tools × 200 files" benchmark. So far we've validated the scoring pipeline end-to-end on one clean-speech condition using an open-source Whisper model as a baseline — VideoText itself has not been run through this pipeline yet, and neither have Deepgram, AssemblyAI, Otter, Descript, or YouTube captions. We're publishing the pilot rather than waiting, because every number here is real, sourced, and reproducible, and we'd rather show partial honest progress than a complete but unverifiable comparison. See the benchmark source code and README for exactly what's measured and what isn't yet.

Phase 1 results (measured)

Metric	Value
Model tested	Whisper baseline — open-source "small" model (faster-whisper, int8, CPU)
Condition	Clean, single-speaker, read speech
Dataset	LibriSpeech test-clean (public domain, professionally aligned reference transcripts)
Utterances / speakers	25 / 5
Audio tested	4.3 minutes
Corpus word error rate (WER)	2.52%
Processing speed	2.9x realtime (CPU, no GPU)

Download the per-utterance results CSV — every row includes the reference transcript, the hypothesis transcript, and the exact error counts used to compute WER.

What's tested vs. planned

Conditions:

Condition	Status	Detail
Clean speech (single speaker)	Measured — Phase 1 pilot	25 LibriSpeech utterances, 5 speakers
Noisy / background-music speech	Not yet measured	Requires a noise-augmented corpus (e.g. MUSAN-mixed speech)
Accented speech	Not yet measured	Requires an accent-labeled corpus (e.g. Common Voice)
Multi-speaker conversation (diarization)	Not yet measured	Requires speaker-labeled ground truth (e.g. AMI, CALLHOME)
Timestamp accuracy	Not yet measured	Requires word-level forced-alignment ground truth

Tools:

Tool	Status	Note
VideoText	Not yet tested	Requires running our production pipeline against the shared dataset
Whisper (open-source baseline)	Tested — Phase 1	2.52% corpus WER on clean speech (this pilot)
OpenAI Whisper API	Not yet tested	Requires an API key
Deepgram	Not yet tested	Requires an API key
AssemblyAI	Not yet tested	Requires an API key
YouTube auto-captions	Not yet tested	Requires captioned video sources
Otter.ai	Not yet tested	No public transcription API — requires manual export
Descript	Not yet tested	No public transcription API — requires manual export

Methodology

Metric: Word Error Rate (WER) — substitutions + deletions + insertions, divided by reference word count — computed via a real Levenshtein-alignment implementation, not estimated. Text is lowercased and stripped of punctuation before scoring (standard practice, since not every engine outputs punctuation).
Ground truth: LibriSpeech test-clean reference transcripts, which are professionally aligned to public-domain LibriVox audiobook recordings.
Reproducibility: every script (dataset prep, transcription runner, scorer, unit tests) is public in the VideoText.io repository, alongside the raw per-utterance results.
Planned additions: Diarization Error Rate (DER) for speaker attribution and timestamp mean-absolute-error are implemented and unit-tested in the scoring engine already, but have not been run yet — they need multi-speaker and forced-alignment ground truth we don't have wired up yet (see Limitations).

Limitations

Small sample: 25 utterances, one condition, one model. Not statistically representative of real-world audio diversity.
Read audiobook speech is easier to transcribe than spontaneous speech, meetings, or phone calls — this pilot's low WER should not be read as a general-purpose accuracy claim.
VideoText has not yet been benchmarked here. When it is, we will publish the result regardless of how it compares to other tools, including any categories where VideoText performs worse.
Diarization and timestamp accuracy are not yet measured against any tool.

Citation

VideoText.io, "AI Transcription Accuracy Benchmark" (Phase 1 pilot), 2026. videotext.io/research/transcription-accuracy-benchmark-2026

Best transcription tool Video to transcript AI transcription tools Industry transcription statistics What is word error rate?