Diagnosing a RAG Metrics Dashboard
Your team's weekly offline eval dashboard for a RAG-based support assistant shows:
- recall@5 = 0.65
- judge correctness (vs. gold answer) = 0.90
- faithfulness = 0.80
Your manager sees the 0.90 correctness number and says "that's a great score, let's ship this to more traffic." Walk through how you would respond: what do these three numbers actually tell you together, what do you suspect is really happening, and what would you fix first before shipping wider? Be specific about what evidence would confirm or rule out your hypothesis.
What the numbers say together
recall@5 = 0.65 means 35% of questions never even get the gold chunk into the candidate set the model sees — that is a hard ceiling on how often the model can be genuinely right for the right reason. Faithfulness = 0.80 means one in five claims in generated answers does not trace back to the retrieved context. Correctness = 0.90 is higher than faithfulness, which should be a red flag rather than reassurance: a correct answer produced by actually using retrieved context should also be faithful by construction, so correctness exceeding faithfulness by 10 points implies a meaningful share of "correct" answers are correct despite not being grounded in what was retrieved.
The hypothesis
On the 35% of queries where retrieval misses, the model is likely answering from its own pretrained/parametric knowledge rather than saying "I don't know." For general, stable facts this can accidentally produce a correct answer — inflating the correctness score — while still being unfaithful, because nothing in the retrieved context supports the claim. This is fragile: it only works when pretraining happens to agree with the actual (possibly proprietary, possibly recently changed) ground truth, which is exactly the content RAG exists to serve.
What confirms or rules this out
Slice both correctness and faithfulness by whether the gold chunk was actually retrieved (recall hit vs. miss) for each query. If correctness on recall-miss queries is close to correctness on recall-hit queries, that strongly supports the parametric-knowledge-cover hypothesis — the model is doing about as well whether or not it had the right context, which it should not be able to do if it were relying on context. If correctness on recall-miss queries collapses toward zero or refusal, the hypothesis is wrong and the 0.90 is a more honest number (though faithfulness overall would then need another explanation, e.g. citation misattribution on hit queries).
What to fix first
Fix retrieval before touching prompts or the judge: better chunking, hybrid dense+lexical search, and re-ranking to close the recall@5 gap. Re-run the full metric suite and expect correctness and faithfulness to converge upward together as retrieval improves. Do not ship wider on the strength of the 0.90 alone — it is likely measuring the model's general knowledge more than the RAG system's actual grounding, and that gap will show up in production as confidently wrong answers on domain-specific or recently changed facts that pretraining does not cover.
Share this question