Allocating a Voice-to-Voice Latency SLO
Product wants a phone support agent that "feels instant", and writes "p95 under 1.5 seconds" in the spec. Your tuned cascade has these assumed stage medians and p95s (milliseconds), measured as the time from the caller's last word to the first agent audio: uplink 90 / 174, endpointing hangover plus jitter 365 / 375, ASR finalization 60 / 137, LLM time to first token 300 / 804, first clause 100 / 193, TTS first byte 120 / 273, downlink 100 / 193. About 30 percent of turns call a backend tool with a 300 to 1,500 ms round trip.
- A colleague adds the p95 column and says the p95 is 2,149 ms; another adds the medians and says 1,135 ms. What is wrong with both, and how do you get the real number?
- A Monte Carlo of these stages gives p50 1,195 ms, p95 1,722 ms and p99 2,124 ms for plain turns. Is the spec met? Which two changes get closest, and what does each cost you?
- How do tool-call turns change the SLO, and what do you do about it?
- Write the SLO you would actually commit to, and say what you measure and where.
1. Percentiles do not add. Summing p95s assumes every stage is at its 95th percentile on the same call, which is far rarer than 5 percent, so 2,149 ms overstates the p95 of the sum. Summing medians ignores the right skew of lognormal stages: the median of a sum of skewed variables exceeds the sum of medians (1,195 versus 1,135 ms here). Use per-stage span data to simulate, or at least convolve the measured distributions; allow for correlation (a shared load factor on the model-side stages widened p95 to 1,908 ms and p99 to 2,466 ms in the simulation); and validate by measuring the end-to-end gap at the client edge. The stage table is still useful as a budget and as ownership: who owns which row, and what lever moves it.
2. Is the spec met? No: p95 is 1,722 ms and 13.4 percent of plain turns exceed 1.5 s. The biggest tail is the LLM (p95 804 ms against a median of 300, a factor e^{1.645 \times 0.6} = 2.68), then the hangover, which is the only large constant. Candidate changes (Monte Carlo, same seed):
| Change | p50 | p95 | Over 1.5 s |
|---|---|---|---|
| baseline: hangover 350, TTFT median 300 | 1,194 | 1,724 | 13.4% |
| TTFT median 300 to 200 (smaller or faster model on simple turns, cached prefix) | 1,093 | 1,472 | 4.2% |
| hangover 350 to 250 | 1,095 | 1,625 | 8.6% |
| both | 993 | 1,374 | 2.3% |
| hangover 500 (safer against cut-offs) | 1,344 | 1,869 | 26.2% |
Cutting the TTFT median alone just meets the p95 target; the hangover change helps the median but costs cut-offs (the endpointing simulation in Speech & Voice Agents gave 40.1, 17.2 and 2.4 percent premature cut-offs at 300, 500 and 800 ms), so pair a shorter hangover with a semantic end-of-turn model rather than a smaller constant. A faster or smaller LLM costs accuracy on hard turns: route simple turns to the small model and measure task success on the routed slice.
3. Tool turns. Without any mitigation a tool turn has silence until the answer: p50 2,390 ms and 82.7 percent over 2 s, and with 30 percent tool turns the whole call has 39.4 percent of turns over 1.5 s and 26.0 percent over 2 s. Have the model emit a short filler clause ("let me check that") before the tool call, so the tool runs while it plays: the initial silence is back to the plain-turn 1,195 ms p50, the gap after the filler has p50 48 and p95 950 ms, and all-turns breach rates fall to 13.6 percent over 1.5 s and 1.6 percent over 2 s. The filler does not speed the tool; if the tool is slow (median 1.1 s) the post-filler gap rises to p95 1,237 ms. Also parallelise independent calls, prefetch the caller's recent orders after authentication, keep tool results short and move anything slower than about 3 s to an asynchronous flow.
4. The SLO. "p50 at most 1.2 s, p95 at most 1.75 s, and at most 2 percent of turns with a silence over 2 s", per call leg, measured at the client edge (last caller speech to first agent audio, plus the longest silence within a turn), reported separately for plain and tool turns, with per-stage spans so a breach can be assigned to a stage. The p95 of 1.5 s is a stretch goal that needs the faster-LLM row. Say that silence is a fraction-of-turns metric, not an average, and that the stage numbers are assumptions until measured on your stack.
Key points. (a) Percentiles do not add; simulate or measure. (b) The LLM tail and the hangover dominate. (c) Fillers convert tool latency into speech but do not speed tools. (d) Commit to percentile and breach-fraction SLOs measured at the edge.
Share this question