Write the Constrained Reward for a Surge Policy
Your team proposes to train a surge-pricing policy that maximizes platform margin per session. You are asked to define the objective and the constraints before any model is built.
- Why is margin alone a dangerous reward for a marketplace?
- Write a reward that includes liquidity, say how you would choose its weight, and say which requirements belong in the reward and which are constraints.
- Name three constraints and how each is enforced in production.
1. Why margin alone is dangerous. A margin-only learner is rewarded for raising the price wherever the extra margin per trip outweighs the trips lost, which pushes towards rationing: fewer, more expensive trips. Rider trust, long-run retention and driver earnings do not appear in the reward, so the learner sacrifices them silently. A short-run margin curve rises long before a retention loss becomes visible.
2. Reward. A workable form per decision is
reward = margin + lambda * completed trips
where lambda is the value of a completed trip beyond this trip's margin, standing for rider lifetime value and driver retention. In the lesson's simulation, margin per trip was 3m minus 1 euros at multiplier m and lambda was EUR 1.00 per completed trip; that lambda is an assumption. Choose it from data: estimate the retention effect of a failed or too-expensive request from a holdout, convert it into the present value of future margin, and then run the policy comparison at several values of lambda to see whether the ranking is stable. If the ranking flips inside the plausible range, that is a business decision to surface, not a modeling detail.
What goes in the reward are tradeable quantities (margin and liquidity). What does not are requirements that must hold regardless of the price you could get for violating them: caps, legal limits and fairness bounds.
3. Constraints and enforcement. (a) A hard multiplier cap (and a lower cap in emergencies): enforced by removing larger actions from the allowed set before the policy picks, in code outside the model. (b) No steep surge in a zone whose supply is already slack (surge cannot draw drivers who are not there): a rule on the allowed set based on a supply or fill-rate feature. (c) A trailing mean-multiplier ceiling or a bound on the share of decisions above a stated rung: a stateful check on the allowed set, and an alert. A trust guardrail on cancel-after-quote rate is not enforced per decision but tested through non-inferiority in experiments and watched in monitoring.
A penalty term would make these tradeable ("the model decides a violation is worth it"), which is what you are trying to rule out. In the simulation, the caps were cheap (about 0.1 percent of profit at a 1.25 cap) because the demand was elastic and the liquidity value penalised rationing; with inelastic event demand or a margin-only objective they would cost more, and the business should hear that.
Share this question