Advanced
Open
Pro
Designing Slim CI for a Slow, Expensive dbt Project
Your dbt project has grown to roughly 400 models. Full CI (dbt build
against the whole project) takes 45 minutes and costs a noticeable
amount of warehouse compute per run, and it runs on every pull request
regardless of how small the change is — a PR that only touches one
staging model's column rename still triggers a full 45-minute, full-
project build. Engineers have started batching unrelated changes into
fewer, larger PRs specifically to avoid paying that CI cost repeatedly,
which is making code review harder and increasing the blast radius of
each merge.
- Design the Slim CI setup that fixes the immediate cost/speed problem, including exactly what state needs to be stored and updated for it to work correctly over time.
- What can go subtly wrong with Slim CI if the stored production state isn't kept in sync with what's actually running in production? Give a concrete failure scenario.
- Slim CI reduces cost per PR, but does it fully replace the value of
periodically running a full
dbt buildagainst the whole project? Justify your answer.
Share this question