Evaluating RAG Systems
"It seems to work" is not an evaluation strategy. A retrieval-augmented generation system can look fine in a demo — the retriever returns something, the model writes a fluent paragraph, the answer sounds plausible — and still be silently wrong for a third of real queries. Free-text generation has no single accuracy number, retrieval failures and generation failures produce the same symptom (a bad answer) through entirely different mechanisms, and a system that is 90% "correct" by one metric can simultaneously be hallucinating on 20% of its claims. Interviewers who ask you to evaluate a RAG system are testing whether you know this, or whether you will reach for a single "quality score" and call it done.
The central discipline this subject teaches is separating retrieval quality from generation quality, because they fail independently and are fixed by completely different levers. If the retriever does not put the right chunk in front of the model, no amount of prompt engineering, few-shot examples, or model upgrade will produce a correct, grounded answer — the information simply is not there. Conversely, a retriever with excellent recall can still feed a model that ignores the context, contradicts it, or pads its answer with claims from pretraining. Interviewers specifically probe this separation because it is the fastest way to tell whether a candidate has actually built and evaluated a RAG system, or has only read about one: ask "your answer quality dropped, what do you check first?" and a candidate who does not immediately say "retrieval recall" has not internalized the layering.
This subject un-compresses the "Evaluating LLM Systems" section of llm-application-system-design specifically for RAG: worked metric arithmetic, gold-set construction mechanics, LLM-as-judge calibration in depth, RAGAS-style pipelines, and hallucination-specific detection techniques. It assumes you know what RAG is (see rag-architecture-end-to-end for the retrieval pipeline itself, and llm-application-system-design for where RAG fits among prompting, tools and fine-tuning). It does not cover production monitoring, drift detection, or online A/B experimentation — that is the scope of llm-observability-and-evaluation; this subject stays inside offline, pre-ship evaluation methodology for RAG specifically.