Comparing Two Rankers With NDCG and MRR

For one search query, the top 4 results from two rankers have these relevance grades (0–3):

• Ranker X: [2, 3, 0, 1]
• Ranker Y: [3, 0, 2, 1]

1. Compute DCG@4 for both using exponential gain (2^rel − 1) and the log2(i+1) discount, then IDCG@4 and NDCG@4.
2. Compute MRR (treating grade ≥ 1 as relevant) and precision@2 for both. Do the metrics agree on which ranker is better?
3. In an offline evaluation over 5,000 queries, X beats Y on NDCG@10 by 0.004. What would you need before recommending X, and why might a real NDCG@10 gain still fail to move click-through in production?