Correcting Position Bias in Click Logs

You have 30 days of search logs: for each search, the impressed listings with their positions and whether each was clicked or booked. A colleague trains a LambdaMART model directly on these logs and reports a big offline NDCG gain, but the online A/B test shows no lift.

Assume examination propensities have been estimated as p = [1.00, 0.70, 0.50, 0.35, 0.25] for positions 1–5.

1. Explain why the offline gain is probably illusory.
2. Describe two concrete methods to reduce the bias in training, and give the inverse-propensity weight of a booking observed at position 4.
3. How would you have estimated p in the first place, and what is the risk of doing so?