Intermediate
Open
Pro
Design a Data Contract Gate for a Feature Pipeline
A feature pipeline produces a table transaction_features consumed by a
fraud model, with columns transaction_id, amount_usd, merchant_category,
customer_tenure_days, and is_flagged_previously. Twice in the last
quarter, a bug in this pipeline reached production undetected by CI:
once when amount_usd started containing negative values after a
currency-conversion bug, and once when merchant_category started
containing an empty string instead of "unknown" for unmapped
merchants, which the downstream encoder silently treated as a new,
rare category.
- Design a data contract for this table that would have caught both past incidents.
- Explain where in the CI/CD pipeline this contract check should run, and why it needs to run in more than one place.
- A teammate argues "we already have unit tests on the pipeline code, isn't that enough?" Explain specifically why unit tests alone would not have caught either incident.
Share this question