MLOps & Production ML

ML System Design Interview Framework

Learn how ML system design interviews are scored, a 7-step framework from requirements to monitoring, back-of-envelope estimation for QPS, embeddings and GPU cost, common problem framings, and the mistakes that sink strong candidates.

ML Data Pipelines & Feature Stores

Design the data side of an ML system: logging for train/serve parity, labelling strategies, point-in-time joins, negative sampling, batch vs streaming features, feature stores, training–serving skew, data validation, lineage and a worked notification-click pipeline.

Data Labeling, Active Learning & Weak Supervision

Learn to design a labeling program with numbers you can reproduce, on a synthetic 100,000-message support inbox with ten imbalanced classes and a 2,000-label budget: why labels set the ceiling (a 20 percent symmetric flip rate cost 0.083 macro-F1 and a class-conditional one 0.127, and a noisy test set misreports even a clean model); agreement statistics implemented by hand and checked against a hand-computed example (Cohen kappa 0.4, Fleiss kappa 1/3, Krippendorff alpha 7/18) with annotators of 85, 80 and 75 percent accuracy reaching only kappa 0.56 to 0.65, honeypot detection power, and a budget in which a USD 0.30 sticker price became USD 1.00 per usable label; pool-based active learning where margin, entropy, uncertainty and a margin-plus-k-means hybrid reached random sampling's 2,000-label macro-F1 with roughly half the labels while pure k-center diversity lost to random, uncertainty sampling bought 35 percent no-signal messages, and evaluating on the AL-selected slice read accuracy 0.432 against 0.664 on a random test set; weak supervision with twelve labeling functions, coverage, precision and conflict tables, majority vote against a Dawid-Skene style label model fitted by EM (similar accuracy, 0.522 versus 0.641 macro-F1), and a correlated-LF failure where four copies of a weak rule drove label-model confidence to 1.000 at 0.218 accuracy; pseudo-labelling and confirmation bias (every threshold failed to beat the supervised baseline here, and rare-class recall fell from 0.160 to 0.033); an explicitly assumption-based LLM-labeling cost table, a simulated distillation and the LLM-labeled-test-set hazard; synthetic data and a measured toy model-collapse demonstration; confident learning by hand with detection precision and recall for injected errors; and versioning, lineage and datasheets. Every measured number comes from fixed-seed scripts on synthetic data; prices and LLM accuracies are labelled placeholders.

Airflow & Workflow Orchestration

A practitioner's tour of Airflow as a data engineering interview topic: how DAGs, tasks, and operators fit together; why the scheduling model's catchup default silently reprocesses history; why classic sensors starve the worker pool and what deferrable operators fix; how retries, SLAs, and alerting are actually wired up; dynamic task mapping for runtime-sized fan-out; why large payloads must never flow through XComs; idempotent task design as the property that makes retries and backfills safe; and a short comparison of Airflow's task-centric model to Dagster and Prefect's asset-centric one.

Data Quality, Testing & Observability

A practitioner's tour of data quality as a data engineering interview topic: the five dimensions of data quality (accuracy, completeness, timeliness, consistency, uniqueness) illustrated with concrete failures, the tradeoffs of enforcing checks at the source, in-pipeline, or at the warehouse, schema contracts between producers and consumers and how to catch breaking changes before they ship, freshness/volume/distribution anomaly monitoring and why it catches what a schema check never will, dbt tests versus dedicated data-observability tooling and when each is the right layer, and a full incident-response walkthrough — triage, containment, root cause, prevention — for a pipeline that silently broke a downstream dashboard.

Containers & Reproducible ML Environments

Learn how Docker images and layers actually work, why ML images balloon to multiple gigabytes, how dependency pinning and lockfiles prevent 'works on my machine' failures that are worse in ML because of native and CUDA dependencies, how to reason about GPU base images and multi-stage builds, and what true reproducibility requires beyond a container: seeding across every layer of randomness and treating data as a versioned pointer, not a copy.

CI/CD for ML & Data Code

Learn what makes CI/CD for ML and data pipelines different from general software CI: unit tests for data transforms, schema and data-contract validation gates, model-quality gates that block a merge on offline metric regression, and continuous deployment of both code and models through staging to production — all built on the core mental model that a model trained on different data is a different artifact requiring the same review rigor as a code change.

Model Training & Experimentation at Scale

Learn how to answer the training half of an ML system design interview: baselines, model family choice, temporal offline evaluation, tuning budgets, experiment tracking, distributed training, GPU cost estimation, embedding tables, retraining cadence, and off-policy evaluation.

Experiment Tracking & Model Registries

Learn how experiment tracking captures runs, hyperparameters, metrics, and artifacts in a tool-agnostic pattern (the MLflow/Weights & Biases model), how a model registry becomes the single source of truth for what's actually deployed, how stage transitions (staging to production to archived) should be gated and by whom, and how to trace a production prediction back through the exact model version, training data snapshot, and code commit that produced it.

MLOps Model Lifecycle & Governance

Learn the complete model lifecycle as a single governed pipeline, why every stage transition needs an owner and a sign-off, how to version data/code/model/environment so any prediction is reproducible months later, and what a model card documents and why regulated teams cannot ship without one.

Responsible ML: Interpretability, Fairness & Privacy

Explain, audit and protect a model before it decides things about people: why regulation (EU AI Act risk tiers, GDPR, fair-lending adverse action) and plain debugging value make this part of the job, the interpretability taxonomy (intrinsic versus post-hoc, global versus local), global methods with their traps (impurity importance ranked a pure-noise feature at 0.103 above a real predictor at 0.053 while permutation importance gave the noise feature -0.001; correlated twins split permutation importance; PDP range 0.12 hiding ICE curves that swing 0.83), local methods (LIME instability across seeds and kernel widths, Shapley axioms, exact brute-force Shapley values matching TreeSHAP to 3e-9 and summing exactly to the prediction, interaction values, counterfactual recourse, Integrated Gradients, Grad-CAM, why attention and LLM self-explanations are not explanations), and turning SHAP into adverse-action reason codes without reading it as causal; fairness (protected attributes and proxies, demographic parity, equal opportunity and equalized odds, predictive parity, calibration within groups, the impossibility theorem demonstrated with numbers, individual versus group fairness, sources of bias, feedback loops), interventions (reweighing, constraints, group thresholds, the four-fifths rule, audits) with a measured trade-off table; privacy (data minimization, k-anonymity limits, differential privacy with the Laplace and Gaussian mechanisms, composition, DP-SGD, federated averaging, memorization in LLMs, synthetic-data caveats, retention); and governance artifacts (model cards, datasheets, risk assessments, human oversight, incident response). The worked example is a simulated credit model with two groups, run in scikit-learn and SHAP, with every number reproduced from code.

Adversarial ML & Model Security

Threat-model an ML system the way a security engineer would and defend the answer in an interview: assets, attacker knowledge and access (white-box versus black-box), a STRIDE-for-ML table and MITRE ATLAS; evasion (FGSM and PGD with an exact linear-model derivation, accuracy-versus-epsilon curves on a 20-dimensional toy task where a logistic regression falls from 96.6 to 82.7 percent at epsilon 0.3 and an MLP to 75.0 percent, transferability to independently trained models, adversarial training that moved MLP robust accuracy from 75.0 to 88.7 percent at almost no clean-accuracy cost until epsilon approaches the signal size, a gradient-masking demonstration where a rounding preprocessor makes a gradient attack report 95.7 percent accuracy while a BPDA attack gets 76.0, and a cost-constrained fraud-feature attack that evades a gradient-boosted model 98.3 percent of the time with three mutable features); poisoning and backdoors (random versus targeted label flips, a 1 percent trigger backdoor with 98.0 percent attack success and unchanged clean accuracy, a spectral-signature filter that only partly worked, provenance filtering that cut attack success to 4.6 percent); privacy attacks (loss-threshold membership inference, AUC and TPR at 1 percent FPR versus generalization gap, output limiting, DP-style training); model theft (extraction fidelity versus query count for logistic and MLP victims, rounding and label-only outputs); supply chain (pickle code execution, safetensors, hub trojans, dependency attacks, SBOM, signing, registry access); serving security including a sponge-input cost multiple measured on an attention-like operation; LLM and agent bridges; governance and incident response; and a worked fraud-API threat model with five mitigations tied to the measured numbers. All attacks run on small synthetic models with numpy, scikit-learn or PyTorch on a CPU; likelihood and impact scores are labeled judgement; library, paper, framework and legal facts are from memory.

Model Serving & Deployment

Learn to design the serving side of an ML system: batch vs online vs streaming vs hybrid inference, latency budgets with numbers, model server patterns, compression, canary and shadow rollouts, feature-store consistency, autoscaling, and safe fallbacks.

LLM Inference Engineering

Reason about how an LLM serving engine works and size one, and defend each choice in an interview: the anatomy of a request (compute-bound prefill versus bandwidth-bound decode, TTFT, TPOT, throughput and goodput, the bytes-per-token latency floor), the KV cache (per-token size formula, a verified cached-versus-recompute toy, the effect of GQA and MQA, PagedAttention fragmentation, prefix caching and KV quantization), batching (static versus continuous in a simulator, chunked prefill and decode stalls, preemption), speculative decoding (acceptance-rate math checked by Monte Carlo, lossless rejection sampling verified empirically, break-even versus batch size), parallelism for serving (tensor, pipeline, expert, prefill/decode disaggregation, multi-LoRA), kernels (FlashAttention, FlashDecoding, CUDA graphs), features that cost latency, the engine landscape, capacity planning and autoscaling signals, observability, and a full worked example for a 70B GQA model at 8k context with 64 concurrent requests. Timing parameters are labeled assumptions; real engines and GPUs were not run.

Model Compression, Quantization & On-Device ML

Shrink a model for latency, cost or a phone and prove you did not break it: where the time and memory go (weights versus activations, memory-bandwidth-bound decode, energy and thermal limits on devices), quantization fundamentals (int8, int4, fp8, NF4 formats, symmetric versus asymmetric affine quantization with scale and zero-point worked by hand, per-tensor versus per-channel versus per-group granularity with measured error, calibration and clipping, int32 accumulation at the matmul verified), PTQ versus QAT (fake quantization and the straight-through estimator, with a 5-seed experiment showing QAT recovers 2-bit accuracy that PTQ loses), LLM quantization (the outlier-channel problem, SmoothQuant, GPTQ and AWQ ideas reproduced as toys, weight-only versus weight-and-activation, NF4 and GGUF families, KV-cache quantization, evaluation beyond perplexity), pruning and sparsity (unstructured versus structured, why zeros in a dense tensor save nothing, N:M 2:4 sparsity, iterative magnitude pruning measured, lottery tickets), knowledge distillation (soft targets, temperature and the T-squared factor verified by gradient, feature and sequence-level distillation, what actually drives the gain), low-rank factorization and LoRA merging (and why merging into int4 weights silently loses the adapter), compilers and runtimes (ONNX export pitfalls reproduced, TensorRT, torch.compile, operator fusion, BatchNorm folding verified), on-device ML (Core ML, LiteRT, ExecuTorch, NNAPI, constraints, the on-device versus server decision matrix, delivery, versioning and on-device A/B), honest benchmarking (p50 versus p99, warm versus cold, batch 1, device variance, accuracy-latency Pareto), and a worked example sizing a 7B model at fp16, int8 and int4 with decode-speed ceilings and an evaluation plan, plus a distill-then-int8-QAT plan for a phone vision model.

Model Release Strategies: Canary, Shadow & Rollback

Learn why a model release is not a normal software deploy because 'correct' is statistical rather than pass/fail, and how shadow mode, canary rollouts, blue/green deployment, gradual ramps, and automatic vs. human-gated rollback triggers manage that risk. Includes a risk-profile-to-strategy decision table and the release metrics to watch during a rollout.

Production Observability & Monitoring

Learn the general observability stack that keeps any production service healthy: the three pillars (logs, metrics, traces), how to build dashboards that answer questions instead of just displaying numbers, SLOs and error budgets, paging policy that avoids alert fatigue, and where ML-specific signals plug into this general foundation.

ML Monitoring, Drift & Retraining

Learn why deployed models decay, how to monitor them in layers from system health to business KPIs, compute PSI and other drift statistics with worked numbers, handle delayed labels, and design retraining triggers with safe validation gates and rollback.

Reliability, Resilience & Observability Patterns

Learn availability math, SLOs and error budgets, timeouts, retries with backoff and jitter, circuit breakers, bulkheads, load shedding, safe deploys, and the three pillars of observability, then apply them to a checkout service with a flaky payment provider.

Cost and Latency Engineering for LLM Apps

The reusable toolkit behind every 'make it cheaper and faster' interview question: why input tokens dominate cost and output tokens dominate latency, how prompt-prefix caching and semantic caching work and where each breaks, model routing and cascades, streaming and parallel tool calls as latency levers, the TTFT-plus-decode model of p95 latency, and the cost-per-1,000-conversations math that ties every lever to a number leadership will ask for.

Case Study: Design a Batch Analytics Platform

Model interview answer for designing the batch analytics platform behind a mid-size company's BI and reporting layer, from OLTP sources through CDC/ELT ingestion, lakehouse/warehouse landing, dbt transformation layers, and Airflow orchestration to dashboards: concrete row-count and byte-volume math that drives every downstream decision, the CDC-vs-batch-extract call made per source rather than uniformly, warehouse-vs-lakehouse and file-format/partitioning choices, staging/intermediate/marts layering in dbt, an Airflow DAG shaped around data-aware dependencies rather than fixed clock time, an SLA built with deliberate slack instead of run at the theoretical minimum, an idempotent partition-scoped backfill strategy, cost control through partition pruning and materialization rather than bigger warehouses, data-quality gates that quarantine rather than silently drop or block, and a team/ownership model that treats the staging layer as a reviewed public interface.

Case Study: Churn Model, End-to-End MLOps

Model interview answer that threads a single churn-prediction model through the full MLOps lifecycle: notebook prototype, CI tests and data-validation gates, experiment tracking and registry promotion, a canary release, a production drift alert, a safely-gated automated retrain, and a governance review before the retrained model ships. Each stage names the specific tool category and the specific failure it prevents.

Case Study: Visual Defect Detection on a Production Line

Model interview answer for industrial visual inspection (electronics, automotive, pharma manufacturing lines, Landing-AI-class vendors): why this is the block's detection-not-generation and edge case, argued against every other case study in the series. The approach ladder from impossible supervised detection to one-class anomaly detection trained on normal parts only — reconstruction-based methods (AnoGAN, f-AnoGAN, grounded in this track's GAN vocabulary) versus pretrained-feature memory-bank methods (PaDiM, PatchCore) — and why memory-bank methods are the current pragmatic default on MVTec-AD-style benchmarks. Golden-sample capture, alignment and lighting normalization, synthetic defects for validation only (cross-referencing this track's synthetic-data-for-CV case), pixel-level anomaly maps pooled into an image-level score, and image-level AUROC plus pixel-level PRO/AUROC as the evaluation pair FID cannot replace. A decision-policy section mirroring the fraud-detection case's asymmetric-cost framing, but for escape cost versus scrap cost. A system design where inference happens on an edge box synchronised to a PLC reject signal, and a scalability deep-dive built entirely around this case's signature problem: one model (or memory bank) per SKU per line, a model-zoo registry, physical drift fixed by re-capturing golden samples, and onboarding a new SKU in hours with 50 images as the real scaling unit.