Budgeting a Guardrail Pipeline Under a Latency Target

Your assistant has a p95 first-token latency target of 800ms. The guardrail checks available to you and their costs:

• Regex PII redaction: ~3ms
• Small-model injection classifier: ~20ms
• Topic/policy filter (cheap classifier): ~15ms
• LLM-as-judge injection classifier (more accurate, catches paraphrased/novel attacks): ~600ms and real per-call token cost
• LLM-as-judge groundedness check on the response: ~700ms and real per-call token cost

Retrieval and the main model call together already consume ~550ms of the 800ms budget.

1. Which checks can run synchronously in the critical path, and which cannot, given the numbers? Show the arithmetic.
2. Design a strategy that still gets the benefit of the two expensive LLM-judge checks without blowing the latency target.
3. An engineer proposes running the LLM-judge injection classifier only when the cheap classifier already flagged something suspicious, "so we rarely pay for it." What's the risk in that design, and how would you mitigate it?