Intermediate
Open
Pro
Reordering a Context Window for Attention and Cache Cost
An engineer on your team assembles the context for each request in this order: (1) the current user message, (2) 8 retrieved chunks, (3) conversation history, (4) tool schemas, (5) the system prompt, placed last "so it's the freshest thing the model reads before answering." The team is debugging two separate complaints: cost is higher than expected for a high-volume, repeat-heavy endpoint, and the model occasionally seems to ignore a specific formatting rule stated in the system prompt.
- Explain why this ordering is a plausible contributor to both complaints, using the distinct mechanisms behind each.
- Propose a reordering and justify each region's new position.
- The engineer pushes back: "if it's near the end, doesn't the model attend to it better, per the primacy/recency point in this material — so shouldn't the system prompt actually stay last?" Address this directly; there's a real subtlety here, not just "no."
Share this question