The Registry Says Nothing Changed. Something Changed.
Your model registry promotes fraud-classifier/v12, storing the model
binary, training metrics, and a content hash of the weights. It does
not version or pin the upstream feature-computation code. Two
weeks later, unrelated to any model work, a data engineer refactors
the feature pipeline — changing how avg_transaction_amount_30d is
computed (e.g., from a calendar-day window to a rolling 30x24h
window). Nothing about fraud-classifier/v12 changes: same version,
same weights, same hash.
What happens to v12's production behavior?
It silently degrades — v12 now scores on features whose distribution no longer matches what it was trained on, with no version change to alert on.
A model registry's version identity — weight hash, training metrics,
lineage of the training run — answers exactly one question: did the
model change? It says nothing about whether the live
feature-computation code feeding that model at serving time still
matches what produced its training data. Feature engineering
typically lives in a separate pipeline (often owned by a different
team, sometimes literally the point of a shared feature store) — a
semantics change there is invisible to the registry. v12 really is,
byte-for-byte, still v12. But the joint distribution of (features,
label) it was trained on no longer matches the distribution of
features it's now being fed — genuine training/serving skew,
introduced by a commit that never touched the model, the registry, or
anything model-related at all.
This is exactly why a healthy-looking registry (same version, same hash, no red flags) is not evidence that served predictions are still valid — model-version tracking and feature-schema tracking answer different questions and neither substitutes for the other. Production ML systems need feature-level schema/distribution monitoring independent of model-version tracking, precisely because the failure mode here produces zero signal on the model-version side. The durable fix is to pin (or at least schema-hash) the feature computation logic alongside the model version in the registry's lineage metadata, so a feature-pipeline change that would break an existing served model is itself a visible, alertable event.
Share this question