Accuracy on Difficult Audio: Noisy, Long, and Technical Recordings

Benchmark accuracy on curated audio looks exceptional. Most real jobs are not curated.

This post covers accuracy when audio is not studio-clean: background noise, accents, cross-talk, technical vocabulary, phone compression, and recordings over 90 minutes.

Why difficult audio loses words

Automatic transcription handles audio in short segments. It does not listen to the full recording and produce one transcript. It processes a chunk, outputs text, processes the next chunk, and repeats.

This matters because:

1. Context is limited to the current chunk. If a speaker introduces a term in minute 10 that clarifies something they said in minute 9, that later context cannot repair the earlier line.
2. Chunk boundaries create artifacts. When speech crosses a boundary, the split can produce duplicated words, dropped syllables, or broken sentences depending on implementation.
3. Speaker tracking is per-chunk unless a separate pass keeps identity. Without that layer, speaker labels do not hold across the whole file.

The tool around the draft is what separates a usable transcript from a frustrating one: how it joins chunks, how it keeps speakers, and how it exports a document you can review.

Test conditions

The same audio samples were run across six difficult conditions. Each test used at least 15 minutes of audio per condition to capture behavior at scale, not just on cherry-picked segments.

Condition 1: Heavy background noise

Test audio: Interview recorded in a coffee shop. Consistent background chatter, espresso machine noise, occasional music. Single speaker, clear diction.

Result: 89.2% word accuracy

What went wrong: The draft followed the speaker's words but occasionally substituted words when background speech or music overlapped the primary voice. The errors were concentrated where background noise peaked.

The practical implication: At 89% accuracy on a 15-minute segment (roughly 2,000 words), you have approximately 220 errors to correct. Background noise is manageable and still adds a real review pass.

Improvement you can make: Recording with a directional mic (even a cheap clip-on lapel) reduces background noise pickup and pushes accuracy back toward the mid-90s on otherwise similar content.

Condition 2: Strong non-native English accents

Test audio: Three separate recordings — one speaker with a strong Indian accent, one with a strong French accent, one with a strong Mandarin accent — all fluent in English, all speaking at a normal conversational pace.

Results:

• Indian accent: 94.1% accuracy
• French accent: 91.8% accuracy
• Mandarin accent: 88.4% accuracy

What went wrong: The errors were not random. They clustered around specific phonemes that the accent modified, particularly vowel sounds and consonant clusters. Some accents are represented more heavily in training data than others, and that shows up in the draft.

The Mandarin accent test had a specific failure pattern: words ending in consonant clusters (like "next," "text," "tasks") were frequently transcribed as the simpler ending that matched the phoneme it heard.

The practical implication: Accent-related accuracy loss is predictable and patterned. Once you know which phonemes cause substitution errors for a speaker, cleanup is faster because you know where to look.

Condition 3: Multi-speaker cross-talk

Test audio: A panel discussion, four speakers, frequent interruptions and overlapping speech, good room audio.

Result: 81.7% accuracy on cross-talk segments; 93.4% accuracy on single-speaker segments within the same recording.

What went wrong: Cross-talk is the hardest problem in transcription. When two voices overlap, the system must choose which voice to follow. The usual behavior is to track the louder or more distinct voice, while the other voice drops out or collides into a nonsense word.

Speaker labels break down in those segments. Diarization is a separate problem from the words themselves, and in cross-talk even a careful speaker pass is unreliable.

The practical implication: If your content has significant cross-talk, plan for a heavy error rate in those segments. Manual correction is often the only reliable option, including re-listening to decide what was actually said.

Condition 4: Technical and domain-specific vocabulary

Test audio: A software engineering interview discussing specific technical concepts, framework names, product names, and company names not in common usage.

Result on proper nouns and technical terms: 68.4% accuracy on the technical vocabulary specifically; 96.2% accuracy on surrounding conversational language.

What went wrong: A general transcription draft was not trained on your product names or proprietary terminology. When it meets an unfamiliar word, it substitutes the closest phonetic match, which for a specific term can be wildly incorrect.

"Kubernetes" appeared as "cue burnettees." A specific product name appeared as three different transcriptions across the recording. A company name the interviewer mentioned repeatedly was transcribed differently almost every time.

The practical implication: Technical vocabulary errors take the longest to correct because they require domain knowledge to spot. They do not look wrong at a glance. They look like real words that happen to be incorrect. This is where human QA is still irreplaceable.

What helps: Building a glossary of proper nouns and technical terms before the listen pass. The guideline format layer lets you document expected terms so the QA pass has a structured reference.

Condition 5: Long recordings (90+ minutes)

Test audio: A 2-hour recorded interview, two speakers, good audio quality, no significant noise.

Result: Accuracy was measured at 15-minute intervals across the recording.

The decline is real and consistent. Over two hours of otherwise clean audio, word accuracy dropped by 3.5 percentage points.

What went wrong: More chunks mean more boundary events and more chances for a split word or a repeated phrase. Voices also drift: how someone sounds in minute 90 is not how they sounded in minute 5.

The practical implication: Do not benchmark a 10-minute sample and assume the same accuracy at 90 minutes. For long recordings, plan a review pass that pays specific attention to the back half.

Condition 6: Audio quality degradation (compression, phone, low bitrate)

Test audio: Three versions of the same 10-minute interview segment — one at full quality (48kHz WAV), one compressed phone call (8kHz telephony codec), one social video export (highly compressed MP4 audio track).

Results:

• Full quality WAV: 96.4% accuracy
• Phone call audio: 83.1% accuracy
• Compressed social video: 91.2% accuracy

What went wrong: Phone audio is the hardest case. Telephony codecs strip high-frequency information the draft uses to tell phonemes apart. With less information, there are more substitution errors.

The practical implication: Phone calls will score lower than video benchmarks. Budget a longer review. If you control the recording, capture at the highest available quality. The difference between a bad recording and a good recording is worth more than switching tools.

Where clean audio still scores well

This post focuses on difficult conditions because that is where the gaps appear. Clean material is a different job:

• Clean, single-speaker English audio: mid-to-high 90s word accuracy is common in practice.
• Natural conversational language: Everyday vocabulary and common idioms are handled well.
• Clear multilingual speech: Languages that are well represented in training data, spoken clearly, stay strong. Translation of an existing subtitle file is a separate tool and is the only place a language count belongs.

Practical accuracy expectations by content type

These are ranges from real-world content, not the optimized sets used in academic benchmarks.

How to reduce cleanup when accuracy falls short

Knowing where a draft fails is the first step. The second is a workflow that keeps the review short when the score is in the lower ranges.

Step 1: Start from a complete draft with speakers and timestamps. A flat text blob makes every later fix harder. Export a document you can mark up.

Step 2: Know your error patterns before the listen pass. Technical vocabulary errors cluster around proper nouns and jargon. Accent errors cluster around specific phonemes. Cross-talk errors cluster around interruption points. A targeted scan beats a character-by-character read.

Step 3: Use a structured QA framework instead of free-form review. A checklist catches more errors in less time than an open-ended read. Check proper nouns first, then speaker labels, then listen to the flagged segments.

Step 4: Document style and formatting rules before delivery. Accuracy is half of transcript quality. Formatting — consistent speaker labels, the right verbatim mode, timestamps, client conventions — is the other half and deserves its own pass.

The bottom line

On clean audio, a modern draft is strong. On difficult audio it can still be useful, and "useful" at 82% on a phone call means a large share of the words need a human.

The benchmark question is not a lab score. It is the score on content that looks like yours. Those numbers are often 10 to 15 percentage points apart, and that gap is the cleanup you will either do yourself or structure so it takes one pass.

The drafts that hold up add speaker tracking and a document shape on top of the words. What you do after the draft matters more than the upload itself.