Evaluation Plan for Swapping the Embedding Model of a RAG System

Your company's RAG assistant indexes 40M chunks (about 300 tokens each) embedded with model A (1,024-d). A vendor offers model B (768-d) with a better score on the MTEB leaderboard and lower price per token. About 30 percent of your queries are non-English, many are short keyword-style queries, and legal wants a guarantee that no document silently disappears from search. The CTO asks: "Can we just switch the query encoder to B and keep the index to save time?" You have two weeks.

1. Answer the CTO's question, with the reason and a demonstration of what would happen.
2. Design the offline evaluation: data, metrics, baselines, slices, and how you decide the difference is real rather than noise.
3. Estimate the cost and time of re-embedding the corpus under stated assumptions, and the index storage difference.
4. Describe the rollout, rollback and monitoring plan.
5. Name three things the MTEB score does not tell you.