Most speech-to-text is benchmarked on audio that looks nothing like a WhatsApp voice note.
The standard evaluation sets are read speech, broadcast news, or recorded interviews: single speaker, decent microphone, one language, quiet room, speaker aware they are being recorded. A WhatsApp voice note is close to the opposite on every axis. I have spent a while building around this, and the gap turned out to be wider than I expected.
Acoustics
Phone held at arm’s length while walking, in a car, in a kitchen, on a street. Distance-to-mic varies wildly within a single recording, which breaks a lot of assumptions about consistent gain.
Then there is the codec. Voice notes are Opus at low bitrate — efficient, but it discards exactly the high-frequency detail that helps disambiguate fricatives. /s/ versus /f/ versus /th/ get genuinely harder, and those distinctions carry real meaning.
Register
Conversational, not read. False starts, self-corrections, filler, trailing off mid-sentence, and long pauses that are not sentence boundaries — someone thinking, or getting distracted.
Punctuation inference is much harder here than on read speech. And punctuation is most of what makes a transcript skimmable rather than a wall of text. A perfectly accurate word sequence with no paragraph breaks is close to useless if the point was to let someone read it faster than listening.
Language
This is the one that surprised me most.
Voice notes are heavily code-switched. People drop English technical terms into Urdu, Hindi, Arabic, Spanish sentences constantly — not as an edge case, as the default register for a huge number of speakers. If you force a single language selection up front, you mangle every mixed utterance.
Auto-detection is not a convenience feature in this domain. It is a correctness requirement.
Length distribution
Most notes are 5–45 seconds. Very little context to work with, and per-request overhead dominates if you architected for long files. Batching strategies that make sense for a 45-minute meeting recording are actively wrong here.
The UX constraint that outranks all of the above
Even a perfect transcript is useless if getting it costs more effort than just listening.
If the flow is export → open another app → upload → wait → copy back, nobody does it twice. I watched this kill the idea in its first version. The transcript has to resolve inside the conversation: share the note, read the text over the chat, keep going.
That single constraint drove more architectural decisions than the model choice did. It rules out anything with a meaningful cold start, anything requiring an account before first use, and anything that needs the user to pick settings before they get a result.
I build HearLess around this — Android, 50+ languages with auto-detect, transcript opens over the WhatsApp chat. It is on Google Play if you want to throw a genuinely awful voice note at it. I would honestly like to know what breaks it.
Curious whether others working on conversational ASR have found approaches to code-switching that hold up in production. It is still the weakest part of what I have, and the literature is thinner than I hoped.
