Long Context vs RAG for a 10M-Token Corpus

1. Why a bigger window doesn't dissolve the problem

Even if a model existed with a 10,000,000-token window that fit the whole corpus: feeding the entire corpus on every turn costs 10,000,000 × $3/1M ≈ $30 per query before output tokens, for even a one-line lookup question — and that's before accounting for the corpus growing over time (new documents are added continuously, so "big enough today" is not "big enough next quarter"). Separately, very long contexts suffer degraded attention to material buried in the middle ("lost in the middle"), so even a model that could fit the corpus would answer less reliably than one given a narrowed, relevant context. Bigger windows are a convenience, not a replacement for narrowing what the model reads.

2. Comparing the three approaches

• Long-context-only (stuff as much as fits every turn): doesn't scale past the window, and even where it technically fits, cost and latency scale with tokens sent on every turn regardless of query complexity.
• RAG-only with small chunks (~300–500 tokens, top-k retrieved): cheap and fast, but fragments context — a clause referencing "the definitions in Section 1" loses that connection — and is weak on synthesis/comparison questions that need several sections read together with their surrounding structure intact.
• Hybrid retrieve-then-long-read: retrieve broadly (recall- oriented) to narrow the 10M-token corpus down to a relevant candidate set, then expand each hit to its full parent section and feed those larger, coherent spans into the model's long-context window in one generation call. This is what I would build: retrieval solves the narrowing problem RAG is good at, long context solves the "read connected material, not disconnected fragments" problem long-context models are good at. Neither alone is sufficient for this corpus and this query mix.

3. Worked example

For the synthesis query, hybrid vector + keyword search returns the top 30 chunks. Each chunk is expanded to its parent section (using the parent-child index built at ingest time) and overlapping/ adjacent expansions are merged, yielding roughly 25 unique sections averaging ~1,600 tokens ≈ 40,000 tokens of retrieved context. That, plus ~2k tokens of system instructions and citation rules and a few thousand tokens of compacted history, comfortably fits the 200k window with headroom for the answer — versus 10,000,000 tokens (50× the window) for the naive full-corpus approach or ~12,000 fragmented tokens from 30 raw 400-token chunks with no surrounding context for a synthesis question that needs it.