Practice — Speech & Voice Agents (6 questions)

You are designing a phone and in-app voice support agent. Product wants "it should feel instant", which you translate into a target of about 1.0 s median and under 1.6 s at p95 for voice-to-voice latency (the user stops speaking to the first agent audio). The first prototype is a cascade: VAD with a 700 ms silence hangover, a non-streaming ASR call after the endpoint, an LLM call, then a TTS call that starts only when the LLM has finished its first full sentence. Users report long awkward pauses.

1. List the stages that sit on the critical path and give a plausible median for each in the prototype and in a tuned design. State clearly that the numbers are assumptions.
2. Show why adding the medians understates the real p50, and why the tail is worse than the stage medians suggest.
3. Which three changes buy the most latency, in order, and what does each one cost in quality or complexity?
4. Under these assumptions can a tuned cascade meet the target? What would you try next?

Your team proposes to put an off-the-shelf Whisper model behind a phone agent because "it has the best accuracy on benchmarks". The agent handles 8 kHz telephone calls in English and Spanish, must show partial transcripts within a few hundred milliseconds, and callers often say product names and order numbers. Silence on the line is common while the caller looks things up.

1. Explain how CTC, RNN-T and an attention encoder-decoder such as Whisper each handle alignment, and which are naturally streaming.
2. List the concrete problems with using Whisper directly in this setting and how you would mitigate each.
3. Recommend an architecture and explain what you would evaluate before committing.