Threat-Model a Video Recommender Before Anyone Proposes a Defense
You are the ML lead for a short-video platform. A ranking model trained on watch time, likes and shares orders the home feed; creators upload videos and anyone can create accounts; partner apps call a public "similar videos" API that returns scored candidates. The security team asks for a threat model of the recommender before the next launch.
- List the assets and the attacker profiles (knowledge, access, goal, budget). Do not start from attack names.
- Walk the trust boundaries with STRIDE and name at least six concrete attacks.
- Rank them by likelihood and impact, and say how honest those scores are.
- Pick the top three mitigations and say what each costs.
- Which parts of this are not this lesson's problem, and where do they go?
1. Assets and attackers. Assets: integrity of the feed (what gets amplified), the model and its embeddings (IP and a stepping stone for crafting attacks), training signals (engagement logs, labels), user privacy (watch history is sensitive), serving capacity, and the pipeline and registry. Attackers, with their knowledge, access, goal and budget (this is the part interviewers probe):
| Attacker | Knowledge | Access | Goal | Budget |
|---|---|---|---|---|
| Promoter or click farm | black-box, sees only the feed | thousands of accounts, can generate engagement | push chosen items up the ranking | modest money, high volume |
| Spammer or policy evader | black-box | uploads, comments | get content past classifiers and ranking | cheap, iterates daily |
| Competitor or scraper | black-box via the partner API | API keys | clone the model or embeddings | can afford many queries |
| Privacy adversary | black-box or outputs only | public outputs, auxiliary data | infer that a person watched something | low |
| Insider or supply-chain attacker | white-box | pipeline, registry, dependencies | persistent compromise | high, rare |
2. STRIDE walk (examples).
- Spoofing: fake accounts and device farms impersonating engagement.
- Tampering: shilling or profile-injection attacks (from memory, the recommender-poisoning term): coordinated accounts inject fake interactions; feedback-loop poisoning of continuous training; tampering with the feature store or a swapped checkpoint.
- Repudiation: no lineage of which run consumed which engagement window, so a poisoning incident cannot be scoped.
- Information disclosure: membership inference from scores, extraction of the ranking or embeddings via "similar videos" (100 queries gave 95 to 99 percent fidelity on the lesson's toys), PII in logs.
- Denial of service: oversized inputs that cost far more compute (1,024 times the attention FLOPs for length 4,096 versus 128 in the lesson).
- Elevation of privilege: loading an untrusted embedding model as a pickle;
trust_remote_codein the training image. - Evasion: perturbed uploads that dodge quality and safety classifiers.
3. Ranking, and honesty about it. Scores below are judgement/ASSUMPTION on 1 to 5 scales, not measurements; write the reasoning next to them and revisit when evidence arrives.
| Threat | L | I | LxI | Reason |
|---|---|---|---|---|
| Engagement poisoning and shilling | 5 | 4 | 20 | cheap, constant, directly rewarded |
| Content evasion of classifiers | 5 | 3 | 15 | cheap iteration; owned by the moderation stack |
| Extraction via partner API | 3 | 3 | 9 | needs budget; value is the embeddings |
| Poisoned or malicious artifact | 2 | 5 | 10 | rare but severe |
| Membership inference on history | 2 | 4 | 8 | regulated data, hard to exploit well |
| Sponge or oversize inputs | 3 | 3 | 9 | easy to try, easy to cap |
4. Top three mitigations and costs. (a) Integrity of training signals: weight engagement by account trust and age, deduplicate and rate-limit interactions per account and per device cluster, detect coordinated behavior (graph methods; see Graph ML & Graph Neural Networks), and gate each retrain on a clean canary set. Cost: lost signal from legitimate new users, more pipeline. The lesson's toy showed that a 1 percent poisoned fraction can implant a trigger with 98 percent success and no accuracy change, so accuracy gates alone are blind. (b) Constrain the API: authenticate partners, per-key query budgets, return ranks or coarse scores instead of raw scores, and monitor probe-like traffic. Cost: product value for partners; rounding alone barely slows extraction, so budgets matter more. (c) Lock the artifact path: safetensors or other non-executable formats, models pulled by digest from a registry only CI can write, signed artifacts, a model SBOM, a kept rollback version. Cost: friction for researchers. Caps on upload size and tokens cover the cheap sponge threat.
5. Out of scope here. Prompt-level attacks if an LLM re-ranks or summarizes: Guardrails and Prompt-Injection Defense. Content-policy evasion and obfuscation arms race: Case Study: Content Moderation System. Architecture and metrics of the recommender itself: Ranking & Recommendation System Architecture and Case Study: Video Recommendation (YouTube / Netflix-style Homepage).
Key points.
- Start from assets and attacker economics, then enumerate attacks; a bare list of attack names is the weak answer.
- In recommenders the training signal is attacker-controlled, so poisoning and shilling lead the ranking.
- Label likelihood and impact as judgement and tie each mitigation to a specific threat, with its cost.
Share this question