Speech & Voice Agents
ASR, TTS and real-time voice pipelines — latency budgets, turn-taking and evaluation
Design and defend a real-time voice agent in an interview: audio basics (sampling and Nyquist aliasing, 25 ms windows with 10 ms hops, a hand-built 80-band log-mel spectrogram, codecs and telephony bandwidth, VAD), speech recognition (CTC versus attention encoder-decoders versus RNN-T for streaming, Whisper's architecture and its hallucination and timestamp limits, WER and CER with a computed normalization experiment where the same eight sentences move from 58.5 to 4.5 percent WER, endpointing, diarization, biasing), text to speech (text normalization, G2P, acoustic model plus vocoder, codec language models, streaming synthesis, SSML, consent, MOS), the voice agent pipeline (cascaded versus speech-to-speech, turn-taking, barge-in, a simulated endpointing trade-off across hangovers of 300, 500 and 800 ms, WebRTC versus WebSocket versus SIP), agent integration under latency (tools, ASR errors, confirmation, DTMF, handoff), evaluation (WER by slice, task completion, latency percentiles, interruption metrics, audio red-teaming), cost and privacy, and a worked latency budget with a Monte Carlo of p50, p95 and p99 voice-to-voice latency for three architectures. All stage latencies, prices and distributions are labeled assumptions; no real ASR, TTS or telephony system was run.
Practice questions (6)
-
View →
Build a Latency Budget for a Sub-Second Voice Support Agent and Say Where It Breaks
Advanced · Free -
View →
Pick an ASR Architecture for a Streaming Phone Agent: CTC, RNN-T or Whisper?
Advanced · Free -
View →
Evaluate an ASR System Fairly Across Accents: Normalization, Slices and Confidence Intervals
Advanced -
View →
Design Barge-In: Interrupting the Agent Without Self-Interruption or Missed Interruptions
Advanced -
View →
Speech-to-Speech Model or Cascaded Pipeline? Decide for a Bank and a Language Tutor
Advanced