Speech & Voice Agents

The boundary. Agent Architectures and the Agentic Loop and Tool and Function Calling cover the reasoning, planning and tool use of an agent; this lesson does not repeat them. Here the subject is the audio modality and the real-time pipeline around an agent: how sound becomes tokens, how tokens become sound, and how you keep the whole loop under a second while a human can interrupt at any moment. Case Study: Voice Customer Service Agent applies this lesson to a full telephony support design. For streaming, TTFT and batching background see LLM Inference Engineering; for the transformer and CTC/seq2seq building blocks see Transformers for AI Engineers and Sequence Models: From RNNs to Attention.

Interviews ask "design a voice assistant that feels instant" and probe for three things: you can do latency arithmetic (a budget, not an adjective), you know why voice is not chat with a microphone (turn-taking, interruption, no screen to recover from errors), and you evaluate with audio-aware metrics, not just a transcript score. Everything that could run on a CPU with numpy was executed with fixed seeds and the output pasted (marked verified; scripts lived in /tmp/speech/). Everything needing a real ASR model, TTS model, GPU or phone network is explained conceptually and marked NOT MEASURED. Product, paper and legal facts are from memory and hedged. Every latency, distribution, price and GPU-concurrency figure is an ASSUMPTION for back-of-envelope reasoning; none is a benchmark of any named product.