Case Study: Content Moderation System
"Design the system that decides which of our users' posts stay up" is a standard prompt at social, marketplace, messaging and generative-AI companies, because it has no single right metric: the cost of a missed threat, the cost of silencing a legitimate user, reviewer capacity, adversaries who change spelling weekly and regulators who ask for explanations all pull in different directions.
This subject is a model answer. It applies theory taught elsewhere instead of re-teaching it: the text pipeline, TF-IDF baselines, calibration and annotator kappa come from Classical NLP & Text Classification; label budgets, honeypots and active learning from Data Labeling, Active Learning & Weak Supervision; group metrics and the impossibility result from Responsible ML: Interpretability, Fairness & Privacy; abstention and routing from Uncertainty Quantification & Conformal Prediction; LLM-side input and output filters from Guardrails and Prompt-Injection Defense; image-plus-text models from Multimodal LLMs and Vision; logging and delayed labels from ML Data Pipelines & Feature Stores; drift monitoring from ML Monitoring, Drift & Retraining; and shadow-testing and holdouts from Advanced Experimentation: Interference, Marketplaces & Platforms. The adversary model is only sketched here; the full threat-model treatment is the Adversarial ML & Model Security. Sibling cases with the same cascade-plus-humans shape are Case Study: Real-Time Payment Fraud Detection (Stripe / PayPal-style) and Case Study: Detecting Deepfakes and AI-Generated Media at Platform Scale.
Every number below is from a synthetic simulation (scripts in /tmp/mod/, fixed seeds, Python with NumPy, scikit-learn and SciPy), not from a real platform. The "text" is made of random pseudo-words standing in for harmful vocabulary, so no harmful content is involved. The strong tier is a simulated stand-in (a noisy score with stated separation), not a measured LLM. Costs, token counts, prices, handling times and appeal rates are labelled assumptions. Read the numbers as showing mechanics and relative sizes, not as benchmarks of any real company. Regulatory remarks are general and hedged: check with counsel.