Intermediate
Open
Pro
Designing Layered Checks for a New Ingestion Pipeline
You're designing a brand-new pipeline that ingests clickstream events
from a mobile app's SDK, lands them in a raw table, and transforms
them into a sessions dbt model used for a product-analytics
dashboard and a churn-prediction feature pipeline. The mobile SDK is
owned by a different team, and you have no ability to add validation
inside the app itself. You have full control over the ingestion
pipeline and the warehouse.
- For each of the three layers (source, pipeline, warehouse), specify exactly what checks you would add given your actual level of control over each layer, and justify each choice.
- The churn-prediction feature pipeline is far more sensitive to subtly wrong data than the dashboard is, since it silently produces bad predictions rather than an obviously wrong chart. How does that change what you prioritize, if at all?
- A teammate proposes skipping dbt tests entirely and relying only on a data-observability platform "since it catches more anyway." Evaluate this proposal.
Share this question