Advanced
Open
Pro
Computing Retrieval Metrics From Raw Results
You are given the top-5 retrieved chunk IDs for four evaluation queries, each with one gold chunk:
| Query | Gold chunk | Top-5 results (ranked, best first) |
|---|---|---|
| Q1 | G1 | A, G1, B, C, D |
| Q2 | G2 | G2, E, F, B, A |
| Q3 | G3 | A, B, C, D, E |
| Q4 | G4 | F, A, B, G4, C |
- Compute recall@5, MRR, and NDCG@5 (binary relevance) by hand.
- Your re-ranker is then changed so that Q1's gold chunk moves from rank 2 to rank 1, and everything else is unchanged. Recompute MRR and explain, in words a non-technical stakeholder would understand, why recall@5 does not change but MRR does.
- A colleague proposes reporting only recall@20 (retrieve a much larger candidate set, don't re-rank) as the system's one retrieval metric going forward. What is missing from that plan?
Share this question