Diagnosing a Two-Week Quality Regression from a Dashboard

1. Why the two symptoms are connected

Faithfulness measures whether every claim in the answer traces back to retrieved context. When retrieval confidence drops below the answer threshold, the system either has thin, marginal evidence to work with or none at all — the model is more likely to either lean on weak context, pad with pretraining knowledge, or produce something that only loosely follows the sources it did get. A rising "no confident source" rate and a falling faithfulness score sharing the same slow two-week slope, with no deploy in between, points at a single upstream cause degrading retrieval quality, not two unrelated bugs.

2. Plausible root causes, in order to check

1. Ingestion falling behind or silently failing — the nightly job or a publish webhook stopped keeping the index current with the source system. New or updated content never lands in the index, so retrieval genuinely has nothing confident to return, and generation follows the same trend because there's nothing solid to ground it in. This is the most common cause of a gradual decline with no deploy, because ingestion pipelines fail quietly (a queue backs up, a webhook silently stops firing) rather than crashing.
2. Query-distribution shift — a product launch, marketing push, or seasonal change is sending a growing share of queries about topics the knowledge base doesn't cover well. Same two symptoms, different fix (content/ingestion coverage, not a pipeline repair).
3. A silent upstream model or embedding version change — if the embedding model or generator is referenced by a provider alias rather than a pinned version, the provider may have rolled a new version behind that alias. This produces a distributional shift with nothing in your deploy log, matching the observed facts exactly.

3. What to pull from traces to confirm

Pull 10–20 of the lowest-confidence recent traces and read: the actual chunk IDs retrieved and their last_updated metadata compared against the source system's latest changes (confirms/rules out ingestion lag); the query topics/intents of the low-confidence requests compared to the historical baseline (confirms/rules out a distribution shift); and the exact model/embedding version fields logged on each trace compared over the two-week window (confirms/ rules out a silent version change). This turns each hypothesis from a guess into a five-minute check, because the trace tree carries exactly the evidence each one needs — that's the entire reason to log chunk IDs, timestamps and versions per span instead of just a final answer.

4. Preventing recurrence

Add an ingestion freshness alert that compares index recency to the source system's latest change (not just "the job returned success"); pin every upstream model and embedding version explicitly and log it per trace so a provider-side change is visible immediately instead of inferred after the fact; and add the confirmed failing cases to the versioned offline gold set so CI would fail a future change (or a scheduled re-run would catch a future recurrence) instead of this being rediscovered from a dashboard two weeks in. This closes the loop between online monitoring, which caught it, and the offline suite, which should catch it earlier next time.