Spotting the Leak in a Target Encoding Pipeline
A colleague encodes a merchant_id column (40,000 distinct values) for a
fraud model like this, run on the full training set before the
train/test split:
merchant_rate = df.groupby("merchant_id")["is_fraud"].transform("mean")
df["merchant_te"] = merchant_rate
Cross-validated AUC on this feature alone is 0.97. In production, AUC drops to 0.71.
- Explain exactly why this code leaks the label, using a merchant with 3 transactions as a concrete example.
- Rewrite the encoding so it is safe to use inside 5-fold cross-validation.
- Why does smoothing (shrinking toward the global rate) matter here even after you fix the leak?
1. The leak:
transform("mean") computes each merchant's fraud rate using all of
that merchant's rows, including the row being encoded. For a merchant
with 3 transactions where 1 is fraud, every one of those 3 rows gets
merchant_te = 1/3 = 0.333 — but the fraudulent row's own label was
used to compute the value it's being given as a feature. For a
merchant with exactly 1 transaction, merchant_te equals that
transaction's label exactly (0 or 1), so the model can memorise "if
merchant_te == 1, predict fraud" with 100% training accuracy and zero
generalisation. With 40,000 merchants likely averaging a handful of
transactions each, most rows get an encoding this leaky, which is why
CV AUC hits an implausible 0.97 and production AUC collapses to
something close to what the feature is actually worth.
2. Safe rewrite (out-of-fold, fit inside the pipeline):
from sklearn.preprocessing import TargetEncoder
from sklearn.model_selection import cross_val_score
# TargetEncoder cross-fits internally: fit_transform() on training
# data uses out-of-fold means, transform() on new data uses the
# full-training-set encoding — exactly the asymmetry you want.
enc = TargetEncoder(smooth="auto")
pipe = Pipeline([("te", ColumnTransformer(
[("merchant", enc, ["merchant_id"])],
remainder="passthrough")),
("clf", model)])
cross_val_score(pipe, X_train, y_train, cv=5, scoring="roc_auc")
Equivalently by hand: split training into 5 folds; for each fold, compute merchant means from the other 4 folds and apply them to this fold. For the final model, compute merchant means from all of training and apply them to validation/test/production rows — those rows were never part of the merchant's own encoding for CV, but the deployed encoding legitimately uses all available training history.
3. Why smoothing still matters:
Even without leakage, a merchant with 2 transactions and 1 fraud gives a raw out-of-fold mean of 0.5 (from a sample size of 1, since one fold is held out) — an extremely noisy estimate that the model will overweight. Smoothing \frac{n_c \bar{y}_c + m\bar{y}}{n_c + m} pulls low-volume merchants toward the global fraud rate, so a merchant with 2 transactions contributes a small, honest nudge rather than a wild swing. It doesn't fix leakage (that's the out-of-fold split) — it fixes the separate problem of high-variance estimates from thin evidence.
Share this question