ML Data Pipelines & Feature Stores

Ask any team what broke their last model in production and the answer is rarely "the architecture". It is a label that arrived late, a feature computed one way offline and another way online, a join that leaked tomorrow's data into today's row, or a schema change upstream nobody noticed. In the ML system design interview this maps directly onto Step 3 (data & labels) and Step 4 (features & pipeline) of the framework from the ML System Design Interview Framework subject — and interviewers weight those steps heavily because they are where real systems fail.

This subject teaches you to design that layer end to end: what to log, how to turn events into labels, how to build a training set that respects time, how to compute features in batch and streaming, what a feature store actually is, and how to keep training and serving consistent. It ends with a worked design — the training set for a "will the user click this notification" model — that you can reproduce on a whiteboard.

Boundaries: model training, hyperparameter search and experiment tracking are covered in the Model Training & Experimentation subject; detecting drift once the model is live is covered in ML Monitoring & Drift. Generic stream-processing and storage infrastructure (Kafka, object stores, partitioning) is assumed from the System Design Interview track.

The Data Flywheel

Production ML is a loop, not a pipeline. The model's outputs change what users see, which changes what gets logged, which changes the next training set.

              ┌──────────────┐        ┌───────────────┐
   users ───► │  serving     │ ─────► │  event logs   │
              │  (model vN)  │        │ (impr, click, │
              └──────▲───────┘        │  features)    │
                     │                └───────┬───────┘
                     │                        │
              ┌──────┴───────┐        ┌───────▼───────┐
              │  model vN+1  │ ◄───── │  labels +     │
              │  (training)  │        │  training set │
              └──────────────┘        └───────────────┘

Two consequences you should say out loud in an interview:

• Selection bias: you only observe outcomes for items the current model chose to show. Without some exploration traffic (a small random or epsilon-greedy slice), the training set never contains evidence that the model is wrong about items it never shows.
• The pipeline is a product: every model improvement is capped by label quality, feature freshness and logging fidelity. Investing in the flywheel compounds; investing in the model alone plateaus.