Designing Confirmation for a Voice Refund
A retailer's phone agent can issue refunds up to USD 100. Today the
model extracts the order number (last four digits) and the amount from
the ASR transcript and calls issue_refund in one step; there is no
read-back. A trial shows a few wrong refunds a week and a manager asks
you to fix it.
Assume per-digit ASR error of 2 percent on the phone line, that a corrupted order id still resolves to a different valid order of the caller 25 percent of the time (otherwise the lookup fails and the agent re-asks), that 30 percent of wrong amounts are rejected by policy, and that after the pipeline has filtered those, 6.9 percent of refund requests proceed with a wrong entity. There are 1,000 refund requests a day.
- Design the confirmation protocol: tools, read-back wording, what counts as consent, and what happens on a barge-in or an unclear answer.
- After a wrong read-back the caller says "yes" anyway 6 percent of the time, and ASR turns a spoken "no" into "yes" 1 percent of the time. Compute the expected wrong refunds per day with no confirmation and with a spoken yes/no confirmation. Which term dominates the residual?
- Active confirmation (the caller repeats the last four digits, or presses a key) lowers the inattentive-yes probability to 2 percent. What is the residual, and what does it cost the caller?
- What do an amount cap of USD 100 and a step-up authentication actually buy, and what do they not?
1. Protocol. Split the action into propose and commit. A
propose_refund(order_ref, amount) tool validates policy (eligibility,
window, amount at most the order total, cap) and returns a proposal id
plus the exact sentence to read; a confirm_refund(proposal_id, confirmation_token) tool commits idempotently and only accepts a
confirmation event bound to that proposal id. The model can therefore
never execute in a single call. Reduce what must be recognised at all:
offer the caller's recent orders ("the blue jacket from March 3?") and
take a full-refund amount from the system, and use DTMF for any digits
that remain.
The read-back names everything the refund depends on: "I can refund USD 42.50 for the blue jacket order ending 4821 to the card ending 1234. Say yes to confirm, or no to change anything." Consent is a fresh-turn yes after the read-back finished. Silence, "uh-huh", a backchannel or a barge-in inside the read-back is not consent; on a barge-in the unspoken part is read again once, and a second unclear answer goes to a human. A "no" or a correction returns to collection; the agent never infers a yes from silence. Validate before reading back, so the safe failures (lookup failure, policy rejection) re-prompt cheaply and the caller is only asked to confirm something the system would accept.
2. Residual risk. A wrong refund needs wrong entities and a yes that is wrong:
with a = 0.06 (inattentive yes) and m = 0.01 (no flipped to yes). So a + (1-a)m = 0.06 + 0.94 \times 0.01 = 0.0694. Without confirmation every wrong entity is executed: 1,000 x 0.069 = 69 wrong refunds a day. With a spoken yes/no: 69 x 0.0694 = 4.8 a day (the Monte Carlo gave 0.475 percent, 4.75 a day), a 14.4x cut. The inattentive caller dominates: ASR flips contribute (1-a)m = 0.0094, about 14 percent of the residual, and the remaining 86 percent is people saying yes without listening. The cut is almost independent of the digit error rate (14.3 to 14.5x from 0.5 to 10 percent in the simulation), because it is set by a and m, so improving ASR lowers the volume of wrong entities but not the confirmation's leverage. Taking digits by DTMF removes the flip and gives 0.411 percent (16.8x), not much better, for the same reason.
3. Active confirmation. With a = 0.02: 0.02 + 0.98 \times 0.01 = 0.0298, so 69 x 0.0298 = 2.1 wrong refunds a day (simulated 0.208 percent, one in 481, 33x versus no confirmation). It attacks the dominant term. The cost: a harder turn for every caller, and a misrecognised "yes" as "no" on 1.5 percent of attempts (a wasted turn on about 1.4 percent of requests in the simulation). Use passive confirmation for small amounts and active confirmation as the amount or risk rises.
4. Caps and step-up. A cap bounds the size of a wrong refund, not its frequency: with spoken yes/no and a USD 100 cap (12.2 percent of requests are above it and go to a human) the probability only fell from 0.475 to 0.425 percent, but expected wrong-refund dollars fell from USD 248 to USD 154 per 1,000 requests (38 percent) and wrong refunds above the cap went to zero. Step-up authentication (a code to the registered device) buys identity assurance against impostors; it does nothing for an ASR error by the genuine caller, so claim it for fraud, not for this problem.
Key points. (a) Propose then commit, bound to a proposal id. (b) Read back item, amount and destination; accept a fresh-turn yes only. (c) The inattentive caller, not ASR, is the residual: use active confirmation as risk rises. (d) Caps bound loss, step-up bounds fraud. (e) All rates here are assumptions; measure yours on real calls.
Share this question