Designing an Empirical Comparison of Two Chunking Configs

Your team is debating two chunking configurations for a new product-docs RAG feature: (A) fixed-size 400 tokens with 15% overlap, and (B) structure-aware chunking capped at 600 tokens with no overlap. Nobody has data, only opinions. You have two days before a decision is needed.

1. Design the smallest evaluation that would give you a defensible answer in two days.
2. What would make you trust config A's result over config B's, or vice versa, beyond a single recall@5 number?
3. Suppose the two configs come out statistically close (0.81 vs 0.79 recall@5). How do you decide, and what do you tell the team?