Paths Subjects Questions Quizzes Pricing Search
Advanced Open Pro

Computing NDCG and Diagnosing a Ranking Regression

A search-ranking model returns 4 results for a query with graded relevance labels (0–3) of the true best order [3, 2, 1, 0]. Your model instead orders them as [2, 3, 0, 1] (i.e. relevance sequence by position: 2, 3, 0, 1).

  1. Compute DCG@4 for the model's order and IDCG@4 for the ideal order, then NDCG@4.
  2. Is this a large or small regression, and why does the position of the error matter more than the fact that an error occurred?
  3. A teammate proposes evaluating this ranker with accuracy (fraction of items in the exact right position) instead of NDCG. What is lost by doing that?

Share this question

← Back to Ranking & Recommendation System Architecture practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.