Case Study: Voice Customer Service Agent

"Design an AI agent that answers our support phone line" is the voice sibling of the most common AI-engineering design prompt, and the voice constraints change the answer more than candidates expect: the caller cannot see a confirmation screen, every second of silence is audible, speech recognition corrupts the very numbers a refund depends on, the caller can interrupt at any moment, and the line is usually recorded. This lesson is a model answer that applies theory taught elsewhere. The audio pipeline, endpointing, barge-in and latency arithmetic come from Speech & Voice Agents; streaming, TTFT and batching from LLM Inference Engineering; the loop and its bounds from Agent Architectures and the Agentic Loop; tool schemas and safe execution from Tool and Function Calling; injection and output controls from Guardrails and Prompt-Injection Defense; spans, judges and online evaluation from LLM Observability and Evaluation. The text predecessor is Case Study: Design a Customer-Support Assistant; this lesson does not repeat its RAG, tenancy or prompt-design material and spends its words on what voice changes. Audio-level attacks (adversarial examples hidden in sound) are only pointed at here; the full treatment is the Adversarial ML & Model Security.

Every number below comes from a fixed-seed Python script (/tmp/voicecs/, NumPy and SciPy, output pasted and re-run) and is a back-of-envelope result on assumed inputs: stage latencies, error rates, prices, handle times and call volumes are labelled ASSUMPTIONS, not measurements of any vendor, model or company. Where a latency distribution reuses the lesson Speech & Voice Agents it says so. Regulatory, PCI and telephony remarks are from memory, not legal advice; check with counsel and your carrier.