Choosing the Ranking Objective for a Two-Sided Marketplace
You are asked to design search ranking for a home-rental marketplace. A product manager proposes: "Rank listings by predicted click-through rate — clicks are plentiful, so the model will train fast." A finance stakeholder counters: "Rank by expected revenue, price times booking probability."
- Explain what goes wrong with each proposal in a two-sided booking marketplace, and state the objective you would actually optimise.
- Write the utility of showing a listing so that host behaviour is included, and say how you would obtain each term.
- Where in the system would you place revenue, diversity and cold-start considerations, and why not in the model's loss function?
1. Problems with each proposal and the right objective
CTR ranking optimises the wrong funnel step. Clicks are cheap and biased toward photogenic or seemingly under-priced listings; many clicked listings are never booked (wrong dates, hidden fees, host declines). A CTR-trained ranker raises engagement while search-to-book conversion stalls or falls.
Expected-revenue ranking (P(\text{book}) \times \text{price}) pushes expensive listings up. Because booking probability falls with price for most guests, conversion drops, and users learn the results are not for them. It also ignores supply health entirely.
The core objective should be a position-debiased booking probability P(\text{book} \mid q, u, l), trained on graded funnel labels (impression < click < wishlist/contact < booking) with a listwise or pairwise learning-to-rank loss, using price as a feature so the model learns price sensitivity rather than being told to prefer high prices.
2. Two-sided utility
- P(\text{request}): the main ranker's output (booking request or instant-book).
- P(\text{accept}): a second head on the same network or a separate model trained on request → accept/decline logs, using host acceptance history, lead time, trip length, guest profile completeness and instant-book flag. For instant-book listings it is 1.
Multiplying keeps guests away from dead ends and rewards responsive hosts.
3. Re-ranking layer, not the loss
Revenue nudges, per-host caps, near-duplicate suppression, and the new-listing exploration boost belong in a versioned re-ranking stage over the top ~300 scored candidates. Reasons: (a) they are policy decisions whose weights are tuned by online experiments, so they must change without retraining; (b) baking them into one loss produces a model no one can debug or attribute regressions to; (c) constraints like "at most two listings from one host on page one" are set-level, which point/pairwise losses cannot express cleanly.
Share this question