Paths Subjects Questions Quizzes Pricing Search
Advanced Open Pro

Designing Slim CI for a Slow, Expensive dbt Project

Your dbt project has grown to roughly 400 models. Full CI (dbt build against the whole project) takes 45 minutes and costs a noticeable amount of warehouse compute per run, and it runs on every pull request regardless of how small the change is — a PR that only touches one staging model's column rename still triggers a full 45-minute, full- project build. Engineers have started batching unrelated changes into fewer, larger PRs specifically to avoid paying that CI cost repeatedly, which is making code review harder and increasing the blast radius of each merge.

  1. Design the Slim CI setup that fixes the immediate cost/speed problem, including exactly what state needs to be stored and updated for it to work correctly over time.
  2. What can go subtly wrong with Slim CI if the stored production state isn't kept in sync with what's actually running in production? Give a concrete failure scenario.
  3. Slim CI reduces cost per PR, but does it fully replace the value of periodically running a full dbt build against the whole project? Justify your answer.

Share this question

← Back to Data Engineering in Production practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.