Fixing p95 Latency by Capping Output Length

Your coding assistant's latency is measured end to end as: retrieval and guardrails pipeline 120 ms, time-to-first-token (TTFT) 350 ms, then decode at 45 tokens/second. The typical (p50) response is 180 output tokens, but occasionally the model dumps a large code block and the p95 response is 900 output tokens.

1. Compute the p50 total latency and the p95 total latency under the current, uncapped setup.
2. You cap max_tokens at 400 and tighten the system prompt to discourage long code dumps, so the effective p95 output length drops to 400 tokens. Recompute p95 latency and the percentage reduction.
3. A teammate instead proposes shaving TTFT from 350 ms to 300 ms by optimizing the guardrail classifiers. Compute the new p95 latency under that change (with output length still uncapped at 900) and explain, using the numbers, why the output cap is the more direct lever for this p95 problem.