Advanced
Open
Pro
Computing NDCG and Diagnosing a Ranking Regression
A search-ranking model returns 4 results for a query with graded relevance labels (0–3) of the true best order [3, 2, 1, 0]. Your model instead orders them as [2, 3, 0, 1] (i.e. relevance sequence by position: 2, 3, 0, 1).
- Compute DCG@4 for the model's order and IDCG@4 for the ideal order, then NDCG@4.
- Is this a large or small regression, and why does the position of the error matter more than the fact that an error occurred?
- A teammate proposes evaluating this ranker with accuracy (fraction of items in the exact right position) instead of NDCG. What is lost by doing that?
Share this question