Choosing a Threshold From a Cost Matrix

A churn model outputs calibrated probabilities. Contacting a customer with a retention offer costs $20 (offer plus agent time). If a would-be churner is contacted, the offer saves them with a value of $150 (net of the offer); if a non-churner is contacted, the $20 is simply wasted. Uncontacted churners are lost.

1. Derive the probability threshold above which contacting is worth it.

2. On validation, the score bands look like this. Compute the total net value at thresholds 0.5, 0.3, 0.1 and identify the best.

   Score band     Churners   Non-churners
   [0.5, 1.0]        400          300
   [0.3, 0.5)        250          500
   [0.1, 0.3)        300        1,800
   [0.0, 0.1)        150        6,300
   

3. Would maximising F1 have found the same threshold? Explain the general relationship between F1-optimal and cost-optimal thresholds.