Diagnosing a Suspiciously Confident LLM-as-Judge

You're evaluating two candidate prompts for a summarization feature using an LLM-as-judge that compares pairs of summaries and picks the better one. Across your 100-example golden set, the judge prefers Prompt B over Prompt A on 78 of 100 examples. Prompt B's summaries are, on average, about twice as long as Prompt A's. You also notice Prompt B was generated by the same model family you're using as the judge, while Prompt A's summaries came from a different provider's model.

1. Name the two specific biases at risk here and explain the mechanism behind each one.
2. Design a corrected evaluation protocol that controls for both.
3. After correcting for both, Prompt B still wins 61 of 100 — how would you decide whether that's now a trustworthy result, and what would make you not trust it?