Build a Latency Budget for a Sub-Second Voice Support Agent and Say Where It Breaks
You are designing a phone and in-app voice support agent. Product wants "it should feel instant", which you translate into a target of about 1.0 s median and under 1.6 s at p95 for voice-to-voice latency (the user stops speaking to the first agent audio). The first prototype is a cascade: VAD with a 700 ms silence hangover, a non-streaming ASR call after the endpoint, an LLM call, then a TTS call that starts only when the LLM has finished its first full sentence. Users report long awkward pauses.
- List the stages that sit on the critical path and give a plausible median for each in the prototype and in a tuned design. State clearly that the numbers are assumptions.
- Show why adding the medians understates the real p50, and why the tail is worse than the stage medians suggest.
- Which three changes buy the most latency, in order, and what does each one cost in quality or complexity?
- Under these assumptions can a tuned cascade meet the target? What would you try next?
1. Stages and ASSUMED medians. Voice-to-voice latency is the sum of first-output times, because everything downstream starts only when the previous stage produces its first usable output. The medians below are assumptions for a back-of-envelope budget (lognormal per stage in a simulation), not measurements of any product.
| Stage | Prototype (ms) | Tuned (ms) | Why it is there |
|---|---|---|---|
| capture + uplink | 50 | 50 | audio buffering plus network |
| endpointing hangover | 715 | 365 | silence needed to decide the turn is over |
| ASR finalization | 299 | 60 | flush, rescoring, punctuation; near zero if streaming |
| LLM time to first token | 501 | 301 | prompt processing plus queueing |
| wait for a usable first clause or sentence | 250 | 100 | TTS needs text to speak |
| TTS first audio byte | 300 | 120 | non-streaming versus clause-level streaming |
| downlink + jitter buffer | 80 | 60 | network plus playout buffer |
| sum of medians | 2,195 | 1,056 |
2. Why adding medians lies. Stage times are right-skewed (a lognormal fits the shape: a long tail from cold caches, queueing, GC, retransmits). The median of a sum of skewed variables is larger than the sum of the medians. In a 200,000-sample Monte Carlo with independent lognormal stages the prototype had p50 / p95 / p99 of 2,325 / 3,297 / 3,979 ms and the tuned cascade 1,106 / 1,629 / 2,037 ms (versus a sum of medians of 1,056 ms). The LLM stage alone has p95 about 2.7x its median for a log-scale sigma of 0.6 (e^{1.645 \times 0.6} = 2.68). Stages are also correlated in reality: a busy LLM provider slows TTFT and often TTS. A shared load factor with sigma 0.3 on the model-side stages moved the tuned cascade to p95 1,811 and p99 2,365 ms. So budget in percentiles, measure per-stage spans in production, and allocate slack to the stages with the fattest tails.
3. Biggest levers, in order.
- Endpointing hangover. Hangover 700, 500, 350 and 200 ms gave tuned p50 of 1,455, 1,256, 1,107 and 956 ms: every millisecond of hangover is a millisecond on every turn. Cost: premature cut-offs. In an endpointing simulation a 300 ms hangover cut off 40.1 percent of synthetic utterances (pauses were an assumed mixture), 500 ms cut off 17.2 percent and 800 ms 2.4 percent. So pair a short hangover with a semantic or prosodic end-of-turn model and context (expected answer type) rather than just lowering the number.
- Stream every stage. Streaming ASR removes about 240 ms of finalization; clause-level TTS with LLM token streaming removes roughly 330 ms (250 to 100 ms of sentence wait and 300 to 120 ms of TTS first-byte time). Cost: prosody at chunk boundaries and more engineering (partial stability, cancellation).
- Speculative generation. Start the LLM on a stable partial transcript before the endpoint is confirmed; if the final text differs, cancel and restart. It took about 70 ms off in the model. Cost: wasted LLM calls (a money cost) and cancellation complexity. Smaller levers: a faster or smaller model for simple turns, shorter prompts and prompt caching (see LLM Inference Engineering), regional deployment close to users, and keeping connections warm.
4. Can the cascade hit the target? Not comfortably. The tuned cascade's median is 1.1 s and only 28.3 percent of turns were under 1.0 s (91.2 percent under 1.5 s); p95 of 1.63 s just misses a 1.6 s target, and with correlated tails it misses clearly. Options: relax the target to something honest ("median about 1.1 s, p95 under 1.8 s"), mask the gap with a short filler or backchannel ("mm-hm", "sure") that plays while the model thinks, route simple turns to a smaller model, or evaluate a speech-to-speech model (assumed median 725 ms, p95 1,125 ms in the same simulation) while accepting its audit and control trade-offs.
Key points.
- Voice-to-voice latency is a sum of first-output times; measure it at the client edge in percentiles.
- The p50 of a sum exceeds the sum of medians; tails multiply when stages are correlated.
- The hangover is the largest single lever and trades directly against premature cut-offs; fix it with a better end-of-turn signal.
- All numbers here are assumptions, to be replaced by per-stage spans from your own deployment.
Share this question