Practice — Case Study: Voice Customer Service Agent (6 questions)

A retailer's phone agent can issue refunds up to USD 100. Today the model extracts the order number (last four digits) and the amount from the ASR transcript and calls issue_refund in one step; there is no read-back. A trial shows a few wrong refunds a week and a manager asks you to fix it.

Assume per-digit ASR error of 2 percent on the phone line, that a corrupted order id still resolves to a different valid order of the caller 25 percent of the time (otherwise the lookup fails and the agent re-asks), that 30 percent of wrong amounts are rejected by policy, and that after the pipeline has filtered those, 6.9 percent of refund requests proceed with a wrong entity. There are 1,000 refund requests a day.

1. Design the confirmation protocol: tools, read-back wording, what counts as consent, and what happens on a barge-in or an unclear answer.
2. After a wrong read-back the caller says "yes" anyway 6 percent of the time, and ASR turns a spoken "no" into "yes" 1 percent of the time. Compute the expected wrong refunds per day with no confirmation and with a spoken yes/no confirmation. Which term dominates the residual?
3. Active confirmation (the caller repeats the last four digits, or presses a key) lowers the inattentive-yes probability to 2 percent. What is the residual, and what does it cost the caller?
4. What do an amount cap of USD 100 and a step-up authentication actually buy, and what do they not?

Product wants a phone support agent that "feels instant", and writes "p95 under 1.5 seconds" in the spec. Your tuned cascade has these assumed stage medians and p95s (milliseconds), measured as the time from the caller's last word to the first agent audio: uplink 90 / 174, endpointing hangover plus jitter 365 / 375, ASR finalization 60 / 137, LLM time to first token 300 / 804, first clause 100 / 193, TTS first byte 120 / 273, downlink 100 / 193. About 30 percent of turns call a backend tool with a 300 to 1,500 ms round trip.

1. A colleague adds the p95 column and says the p95 is 2,149 ms; another adds the medians and says 1,135 ms. What is wrong with both, and how do you get the real number?
2. A Monte Carlo of these stages gives p50 1,195 ms, p95 1,722 ms and p99 2,124 ms for plain turns. Is the spec met? Which two changes get closest, and what does each cost you?
3. How do tool-call turns change the SLO, and what do you do about it?
4. Write the SLO you would actually commit to, and say what you measure and where.