Case Study: Content Moderation System
Tiered classifiers, LLM review and humans in the loop under adversarial pressure and regulation
Model interview answer for a user-generated-content moderation system (5 million posts a day, about 0.5 percent harmful across four policy categories with severity tiers): turn policy into labels and a cost matrix, derive closed-form allow/review/remove thresholds, build a cheap TF-IDF tier, a stronger simulated embedding/LLM tier and a human queue, route by expected loss instead of raw score and compare the routers at equal strong-tier budget, size reviewers and queues with Little's law and Erlang C, measure recall by stratified prevalence sampling of the allowed bucket (naive versus stratified, with sample-size arithmetic), measure precision at enforcement and the appeal overturn rate (and why it is not 1 minus precision), watch adversarial obfuscation drift and recovery by normalisation, audit dialect false-positive disparity at the enforcement threshold, trade pre- against post-publication latency, control LLM-tier cost with explicit assumptions, and close with retraining cadence, dashboards, hedged regulatory notes and reviewer wellbeing. All numbers come from a fixed-seed synthetic simulation with abstract pseudo-word 'harm'.
Practice questions (5)
-
View →
Deriving Allow, Review and Remove Thresholds from Costs
Advanced · Free -
View →
Measuring Recall When You Cannot See What You Missed
Advanced · Free -
View →
A New Scam Format Appears Overnight
Advanced -
View →
Designing a Dialect Fairness Audit That Detects the Real Problem
Advanced -
View →
Controlling the Cost and Capacity of an LLM Review Tier
Advanced