LLM Observability and Evaluation

You cannot debug a system you cannot see, and you cannot improve a system you cannot measure. That sounds obvious for any production service, but LLM systems break the tooling most engineers already trust to see and measure. A traditional API fails loudly: a 500, a stack trace, a latency spike, a failed assertion. An LLM system can fail in complete silence — it returns a 200, the JSON is well-formed, the answer is fluent and confident, and it is simply wrong. Nothing throws. Nothing pages. The only symptom is that a support conversation didn't actually resolve the customer's problem, or a coding agent quietly deleted the wrong test, or a "faithfulness" number that nobody was watching drifted down by ten points over three weeks because a knowledge source went stale. Silent quality drift is the defining operational risk of LLM systems, and it is exactly the failure mode that conventional APM (latency, error rate, uptime) does not catch.

This subject is about the discipline that catches it: instrumenting every request deeply enough to answer "why did it say that" after the fact, treating cost and latency as product metrics rather than infra footnotes, and running two complementary evaluation loops — a deterministic, versioned offline suite that gates every change before it ships, and an online monitoring loop that watches live traffic for the drift the offline suite didn't anticipate. It also covers the technique that makes both loops affordable at scale — LLM-as-judge — including the biases that make it dangerous to trust blindly, and the operational practices (CI regression gates, canary rollouts) that turn "we evaluated it once" into "we cannot regress without knowing."

Two sibling subjects go deeper on pieces of this picture and are referenced rather than repeated here. evaluating-rag-systems covers the RAG-specific offline metrics — recall@k, MRR/NDCG@k, RAGAS-style pipelines, hallucination-detection techniques — in far more depth than this subject needs; this subject stays at the level of "offline evals as a general practice" and sends you there for retrieval-metric arithmetic. ml-monitoring-and-drift covers general monitoring and drift theory — PSI, concept drift, delayed labels, retraining triggers — that applies to any deployed model, LLM or not; this subject covers only what is specific to generative, free-text systems. llm-application-system-design introduces tracing and evaluation as part of the broader system-design picture; this subject un-compresses its "Observability" section into a full treatment. And case-study-coding-agent has a worked observability section for a specific system — a terminal coding agent — that is a good concrete companion to the general treatment here.