Pick an ASR Architecture for a Streaming Phone Agent: CTC, RNN-T or Whisper?
Your team proposes to put an off-the-shelf Whisper model behind a phone agent because "it has the best accuracy on benchmarks". The agent handles 8 kHz telephone calls in English and Spanish, must show partial transcripts within a few hundred milliseconds, and callers often say product names and order numbers. Silence on the line is common while the caller looks things up.
- Explain how CTC, RNN-T and an attention encoder-decoder such as Whisper each handle alignment, and which are naturally streaming.
- List the concrete problems with using Whisper directly in this setting and how you would mitigate each.
- Recommend an architecture and explain what you would evaluate before committing.
1. Alignment and streaming.
| Family | Alignment | Language model | Streaming |
|---|---|---|---|
| CTC | a blank token plus a many-to-one collapse; loss sums over all valid alignments | none internal (outputs conditionally independent given the audio), so add an external LM or shallow fusion | yes with a causal or chunked encoder |
| RNN-T | the joint network scores (frame, previous output tokens) and emits a token or blank at each frame, possibly several tokens per frame | internal, via the prediction network over previous tokens | yes, the standard streaming and on-device choice |
| Attention encoder-decoder (Whisper) | the decoder cross-attends to the whole encoded window and generates autoregressively | strong implicit LM | no; it processes a window (30 s in Whisper, from memory) |
Streaming in CTC and RNN-T means a chunked or causal encoder with a limited look-ahead; the look-ahead is paid as latency, so you tune chunk size against accuracy. RNN-T's cost is training memory (the joint tensor is batch by frames by tokens by vocabulary), usually managed with pruned or memory-efficient loss implementations.
2. Problems with raw Whisper here (from memory of its design).
- No native streaming. Decoding a growing buffer repeatedly wastes compute and makes partials flicker; chopping into fixed chunks cuts words at the boundaries. Mitigation: use a streaming model for the live path, and use Whisper (if at all) for post-call transcription or as a verifier for risky turns.
- Hallucination on silence and noise. A decoder conditioned weakly on audio produces fluent text for silence: for a reference of "turn on the lights" (4 words) plus a hallucinated tail of eight extra words, WER is 8/4 = 200 percent (computed). Mitigation: VAD-gate audio so silence never reaches the model, use no-speech and compression-ratio / log-probability thresholds with fallback, disable conditioning on previous text, and test explicitly on silence, hold music and noise.
- Telephone bandwidth mismatch. 8 kHz G.711 audio lacks the 4 to 8 kHz band that wideband-trained models rely on, and upsampling does not restore it. Mitigation: evaluate on real post-codec call audio and fine-tune or adapt on telephony data.
- Language ID errors on short utterances. A one-word reply may be detected as the wrong language. Mitigation: constrain the language set from caller metadata, or run language ID over a longer window and carry it across the call.
- Domain terms and numbers. Product names and order numbers are rare tokens. Mitigation: contextual biasing (a phrase list or an initial prompt), plus a catalog lookup that re-ranks hypotheses, and spoken read-back of entities at the dialogue layer.
- Timestamps are approximate; use forced alignment if word times matter.
3. Recommendation and evaluation plan. For the live call path: a streaming transducer or chunked-CTC model with endpointing, a small neural VAD in front, contextual biasing, and the language chosen per call. Keep a Whisper-class model offline for QA transcripts and as an adjudicator. Before committing, run an A/B on a held-out set of real 8 kHz calls with accents, noise, speakerphone and silences, scoring:
- normalized WER with a published normalizer, sliced by language, channel and noise, plus entity error rate on names and numbers;
- partial-transcript latency (time from word spoken to word shown) and finalization delay, as percentiles;
- hallucination rate on silence-only segments (insertions per minute);
- cost per concurrent stream and GPU density under load.
Choose the cheapest model that meets the entity-accuracy and latency targets on your slices, not the leaderboard winner.
Key points.
- CTC is simple and fast but has no internal LM; RNN-T has one and is the streaming workhorse; Whisper-style models win offline but are not natively streaming and can hallucinate.
- Gate the model with VAD, bias for domain terms, and test on real narrowband audio.
- Pick by task metrics and latency on your own slices.
Share this question