Graph Features or a GNN for Telecom Churn?
A mobile operator has 2,000,000 subscribers, a monthly churn rate of 3 percent, and a call graph (who calls whom, with call counts and timestamps). A data scientist proposes "a GraphSAGE model on the call graph" to predict next-month churn, because churn is known to spread through friend groups. The current model is a LightGBM on billing and usage features.
- Before any model, how would you check the claim that churn spreads along the call graph? Compute the edge homophily expected by chance at a 3 percent churn rate and explain why the raw edge homophily is the wrong statistic to report.
- Propose a baseline ladder from the current model to a GNN, naming the graph features at each rung.
- Name two leakage traps in a feature such as "share of this subscriber's contacts who have already churned".
- State the decision rule you would use to accept or reject the GNN.
1. The homophily check. With 3 percent positives, two random endpoints share a label with probability 0.97^2 + 0.03^2 = 0.9418. So an observed edge homophily of, say, 0.95 sounds high yet is barely above chance (verified arithmetic). Report instead the class-conditional neighbour share: among contacts of subscribers who churned, what fraction churned in the same period, against the 3 percent base rate. A share of 9 percent would be a lift of 3 and supports the claim; a share of 3.5 percent would not. Also look at the share by contact strength (calls per month) and compare with a shuffled-edge control: randomly rewire the graph keeping degrees and recompute. If the real graph is not clearly different from the rewired one, there is no graph signal for any model to find. Do this on a time-ordered basis: neighbours' churn in month t against the subscriber's churn in month t+1.
2. Baseline ladder. (a) The current LightGBM. (b) Add structural features computed on the call graph snapshot before the scoring date: degree, weighted degree (call minutes), reciprocity, PageRank, clustering coefficient or triangles, and community size and density from Louvain (never the community id). (c) Add neighbour aggregates: the mean and maximum of contacts' usage and billing features, the share of contacts on a competitor network, 1-hop and 2-hop. (d) Add the as-of neighbour label feature only if labels are available at scoring time (see 3). (e) Only then a 2-layer inductive GraphSAGE using the same inputs, trained on a past snapshot and scored on the latest one. Each rung is a small, cheap experiment and the gain from rung to rung tells you what the graph is worth.
3. Leakage traps. (i) Time: "contacts who have churned" must count only contacts whose churn was known before the scoring date. Using the final label of every contact includes contacts who churn after the subscriber does, which is the prediction target leaking backwards; in the lesson's synthetic fraud example the analogous mistake lifted PR-AUC from 0.508 to 0.912. (ii) Simultaneity: when a household churns together, the contact's churn and the subscriber's churn are one event, so a feature that is "known" a day earlier is still not available in production if the contact's status is recorded with a delay. Define the feature as of the same cut-off the model will use at serving time.
4. Decision rule. Train on a past snapshot, evaluate on a later period with several seeds, and compare PR-AUC (or lift in the top decile, which is the quantity the retention team acts on) of the GNN with the best rung of the ladder. Accept the GNN only if the gain exceeds the seed-to-seed spread (in the lesson's synthetic example the standard deviation across 3 seeds was 0.04 to 0.08 PR-AUC, so a 0.02 gain would be indistinguishable from noise) and if it is worth the extra cost of a batch embedding job and monitoring. If it ties the features, ship the features.
Share this question