Diagnosing 'Plausible but Wrong' Answers

1. Root causes, walked through the pipeline

"Confident and subtly wrong" — as opposed to garbled or off-topic — points specifically at retrieval returning a plausible-but-wrong chunk (a similar plan, a different product version, an adjacent feature) that the model then answers from fluently, plus nothing downstream catching it. With no re-ranking, raw cosine similarity from a single dense pass decides what reaches the prompt, and cosine similarity is a coarse signal — it can rank "the Pro plan's price" and "the Enterprise plan's price" close together because they're topically similar, even though only one answers the question. With no confidence gate, the system always answers from whatever it retrieved, so there's no mechanism to say "these results aren't good enough." Fixed-size chunking is a secondary suspect — it can also merge a caveat with the wrong surrounding section — but the described symptom (a specific wrong value, not a garbled non-sequitur) is more consistent with a wrong-but-fluent chunk being retrieved and trusted than with a mid-sentence split.

2. Ordered changes

1. Hybrid retrieval + re-ranking first. Add a BM25 index merged with dense results (catches exact tokens like plan names, SKUs, version numbers that embeddings blur), and add a cross-encoder re-ranking pass on the top-20/30 candidates before picking the top-n for the prompt. This directly targets "plausible but wrong beats actually correct" — re-ranking scores query and chunk jointly, which is far better at distinguishing "similar plan" from "right plan" than embedding cosine similarity alone.
2. Confidence gate. Once re-ranking produces a real relevance score, gate on it: below threshold, don't answer from the KB — say so or hand off. This won't fix wrong retrievals but it stops the system from presenting a low-confidence retrieval as a confident answer.
3. Citations + a post-check. Require numbered citations and verify cited ids exist; this doesn't prevent the failure but makes it auditable — a reviewer (or an automated faithfulness check) can now see which chunk the wrong claim came from.
4. Chunking/metadata review, lower priority here: add metadata (product, plan tier, version) and filter on it at retrieval time so a query about the Pro plan structurally cannot retrieve an Enterprise-tier chunk, regardless of how similar the text reads.

The order is deliberate: fix the stage most likely causing this symptom first (retrieval quality), then add the safety net that makes future instances visible (gate + citations), then harden the structural cause (metadata filtering) rather than starting there.

3. Making it visible in production

Add an offline eval slice specifically of "near-miss" cases — pairs of genuinely similar but distinct entities (plan tiers, product versions) — and track recall@5 and re-rank precision on that slice specifically, since it will look fine on average recall while failing exactly this pattern. Online, track the confidence-gate firing rate (before this fix, it was firing 0% because there was no gate — that alone is a red flag) and sample citations for a faithfulness check on a daily rolling basis, flagging claims that don't hold up against their cited chunk. Route those flags back into the offline gold set so the same near-miss pattern doesn't regress silently later.