Intermediate
Open
Pro
Data Validation Gates and a Feature Backfill
Your nightly training pipeline for a recommendation ranker has three
stages: (a) build labels, (b) assemble features via the offline
store, (c) write the training set. On Tuesday, an upstream team
changed the device_type enum from {ios, android, web} to
{iphone, ipad, android, web} without notice, and separately your
team wants to change the definition of user_ctr_30d to exclude bot
traffic.
- Which validation checks would have caught the enum change, and what should the pipeline do when they fire?
- Explain how to roll out the new
user_ctr_30ddefinition without creating a training window that mixes both definitions. - What property must the feature-materialisation job have for the rollout in (2) to be safe to retry, and why?
Share this question