Computing and Interpreting NDCG@5 for Two Rankers

For one search session the ground-truth grades are: booked = 3, wishlisted = 2, clicked = 1, ignored = 0. Use exponential gain 2^{rel} - 1 and discount \log_2(i + 1).

Ranker A returned listings with grades [1, 0, 3, 0, 0]. Ranker B returned listings with grades [0, 3, 1, 1, 0].

1. Compute DCG@5, IDCG@5 and NDCG@5 for each ranker.
2. Which ranker is better by NDCG@5, and which by MRR of the booked listing? Explain any disagreement.
3. Why should you not stop at these offline numbers before shipping?