Defend a Weekly-Retrained Fraud Model Against Feedback Poisoning
A payments fraud model is retrained every week on three label sources: chargebacks (arrive 30 to 60 days late), analyst decisions, and merchant "this was legitimate" feedback submitted through an API. A red team shows that a small group of colluding merchants can submit crafted feedback so that a specific transaction pattern is later scored as safe, while ordinary test metrics do not move.
- Explain the attack in terms of what the attacker controls, and why the usual offline accuracy gate does not catch it.
- Design defenses at four layers: data intake, training, promotion gate, and production. Say what each costs.
- Which of the defenses did the lesson's toy experiments support or fail to support, and what does that mean for how you rely on them?
- Write the promotion-gate checklist you would actually enforce.
1. The attack. The attacker controls a source of labels, not the model. The merchants' feedback is an unauthenticated-in-practice training signal, so they can implement a backdoor (a pattern plus a "legitimate" label on a small number of rows) or a targeted label flip (flip one class in one region). The model keeps its general behavior because the poisoned rows are a tiny fraction; in the lesson's toy, 1 percent poisoned rows gave 98.0 percent attack success with test accuracy 0.961 versus 0.958 for the clean model, and 0.5 percent still gave 80.5 percent. An offline accuracy gate on a random holdout measures the average, and the average is not what the attacker changed. Random label noise is mostly harmless (accuracy 0.964 at 20 percent random flips) while class-targeted flips are damaging (class-1 recall fell to 0.759 at 30 percent flips of that class), which is why the kind of corruption matters more than its amount.
2. Layered defenses.
| Layer | Control | Cost |
|---|---|---|
| Intake | authenticate and score each source; per-source and per-account rate limits on feedback; weight sources by trust (chargebacks over merchant opinions); sign and version batches | slower learning from the least-trusted source |
| Intake | source-level provenance: every row carries source id, so a bad source can be dropped wholesale | storage and schema discipline |
| Intake | outlier and distribution checks per source (feature ranges, label rate by segment, duplicates) | catches loud attacks only |
| Training | robust options: sample trimming by loss, source-balanced sampling, ensembles trained on different source subsets, DP-style clipping | accuracy and complexity |
| Promotion | human-verified canary set (never scraped, refreshed by analysts), slice metrics, trigger and pattern scans, behavioral diff versus the previous production model on a fixed probe set | compute; analyst time |
| Production | shadow or canary rollout, monitoring of score distribution by segment, instant rollback to the previous registry version | needs a kept, tested rollback path |
3. What the toy supported and failed to support.
- Provenance filtering worked. Dropping the single untrusted source (400 rows: 80 poisoned plus 320 clean, 8.2 percent of the clean data) cut attack success from 0.996 to 0.046 with clean accuracy 0.959. The condition is that the source is identifiable; that is a data engineering property, which is why intake provenance is the first row in the table.
- The spectral-signature filter failed to deliver. At 1, 2 and 5 percent poison it removed only 30, 45 and 59 percent of poisoned rows while deleting 2.7, 5.2 and 12.7 percent of clean rows, and attack success after retraining was unchanged (0.996 before and after at 2 percent). A 2-means split of the class representation contained all 80 poisons but 828 clean rows too. The lesson: a published defense is a hypothesis about your representation; measure removal recall, false-removal rate and post-retrain attack success before relying on it.
- A range check caught only the loud trigger. A threshold at absolute value 4 flagged 80 of 80 poisoned rows (0 of 3,920 clean) when the trigger value was 5.0 and 0 of 80 when it was 2.0, where the attack still succeeded 93.2 percent of the time. Input rules defend against the attacker you imagined.
4. Promotion-gate checklist (thresholds are ASSUMPTIONS to tune).
- Candidate data manifest lists every source, its row count and its label rate by segment; any source whose label rate shifts by more than a set tolerance versus its history blocks promotion until reviewed.
- Canary accuracy and recall within a fixed tolerance of the current production model (for example no worse than 0.5 point on the aggregate and 1 point on each protected slice).
- Behavioral diff on a frozen probe set including known evasion patterns: the share of probes whose decision flips from "block" to "allow" must stay under a small threshold, and every flip is reviewed.
- A trigger scan: perturb canary fraud rows with candidate patterns (rare values, unusual feature combinations found in the newest feedback) and check the score does not collapse.
- Registry promotion needs a reviewer other than the pipeline owner, the artifact is signed, the data snapshot is pinned, and the previous model stays deployable for rollback (see Experiment Tracking & Model Registries and MLOps Model Lifecycle & Governance).
- Post-promotion: staged traffic, monitoring of score and approval rate by merchant segment, and a pause switch for the retraining job (see ML Monitoring, Drift & Retraining).
Key points.
- Poisoning through feedback is a data-source trust problem first and a modeling problem second.
- Accuracy gates cannot see a backdoor; use source provenance, canaries and behavioral diffs.
- Measure every defense on your own model; the toy's spectral filter left the attack intact while provenance removed it.
Share this question