Diagnosing a Dashboard That Broke Days After the Actual Cause

1. Why every existing check passed

This is a semantic schema change with no structural signature: the field kept its name (plan_price), kept its type (numeric), and remained non-null on every row. A schema validator checks structural properties — field presence, name, type, nullability — and every one of those properties is unchanged. A NOT NULL constraint has nothing to object to, since the field is still populated. unique isn't even relevant to this field. The actual break is in meaning: the number now represents a different unit (dollars instead of cents), and no mechanical, structure-based check can detect a meaning change that leaves every structural property intact. This is precisely the category of break that requires either an explicit value-level contract (e.g., a documented, tested assumption about the field's unit or expected range) or a check that has memory of the field's historical values to compare against — which is exactly what part 2 builds.

2. Monitoring that would have caught it within a day

A distribution monitor on plan_price — tracking the mean, min, and max per daily load, compared against a rolling historical baseline — would have fired on Wednesday's very first load: a field whose average value drops by roughly 100x overnight is an obvious outlier against any reasonable historical band, even a simple standard-deviation threshold. This works precisely because it doesn't require anyone to have anticipated "the billing provider might change units" in advance; it only requires noticing that a numeric column's aggregate shape moved sharply, which is what distribution monitoring is built to do independent of the specific cause. As a second, complementary check: a dbt reconciliation test comparing SUM(monthly_recurring_revenue) computed in the warehouse against an independent total pulled directly from the billing provider's own reporting endpoint, run on every build, would fail the build itself before the dashboard ever refreshes with the bad number — catching it even earlier, at the pipeline layer, rather than after the data is already exposed to a dashboard.

3. Incident response walkthrough

First, triage: before touching anything, confirm the scope — query when the drop actually started (Wednesday's load, not a gradual decline, which points to a discrete upstream change rather than slow drift) and trace lineage to see what else depends on plan_price. In this case, only the MRR model is affected; other dashboards built on the same source table but not referencing price (signup counts, churn) are untouched, so the blast radius is one metric, not the whole table. Second, containment: mark the MRR dashboard as stale with a visible banner rather than leaving a confidently wrong (99%-lower) number up where an executive might report it externally, and pause the specific model consuming plan_price so Friday and Saturday don't compound three more days of bad data on top of the three already affected. Only third, root-cause: diff a captured API payload from before and after the change to confirm the exact mechanism (cents-to-dollars unit change). The order matters because root-causing first, before containment, would mean spending time investigating while the dashboard keeps showing a wrong number that people might act on — stopping the damage takes priority over understanding it.

4. Longer-term fix and why monitoring alone isn't enough

Open a conversation with the billing provider about a versioned API contract with a deprecation-notice process for breaking field changes — the underlying problem is that an externally-owned producer can change the meaning of a field with zero warning, and no amount of monitoring on your side prevents that from happening again on some other field next quarter. Monitoring (part 2) is necessary but not sufficient on its own because it's reactive — it shortens detection time from days to hours, but it doesn't prevent the underlying risk, which is a producer relationship with no contract. The systemic fix addresses the category of risk (any field from this producer could silently change meaning); the monitoring fix addresses the speed of detecting the next instance of it. A mature postmortem does both, and explicitly separates "what we fixed" from "what we prevented" so the team doesn't mistake a faster detection mechanism for having removed the risk.