A Metric Changed Meaning and Nobody Downstream Was Told
A dashboard used by the sales team shows "active customers" dropping
15% week over week, triggering panic in a Monday leadership meeting.
Investigation traces it to a dbt model change merged the previous
Thursday: an analytics engineer redefined is_active_customer to
exclude accounts with a canceled-but-not-yet-expired subscription,
because a different team (finance) needed that stricter definition for
a churn report and asked for the change in a Slack DM. The PR passed
CI (all existing tests still passed, since the tests checked the
column exists and is boolean, not what specific accounts it includes)
and was reviewed by another analytics engineer who wasn't aware of who
else consumed the model.
- Diagnose the actual root cause here — not "the definition changed" but why a change requested by one team was able to silently break another team's dashboard with no warning.
- Design the process and technical changes that would have caught this before merge, or at minimum, warned the sales team before Monday's meeting.
- Finance's need (a stricter definition for churn) and sales's need (the original, looser definition for their dashboard) are both legitimate. What's the right long-term modeling fix, rather than picking one team's definition as the sole "correct" one?
Share this question