Diagnose a Bi-Encoder Fine-Tune: What Temperature Does to InfoNCE
You fine-tune a product-search bi-encoder (query tower and listing tower) with in-batch InfoNCE on (query, clicked listing) pairs. With temperature \tau = 1 the training loss plateaus near 0.6 and recall@10 barely moves. Lowering to \tau = 0.02 makes the loss go almost to zero within a few hundred steps, but offline recall on a held-out set gets worse and the team notices the data contains many duplicated listings (the same product from several sellers).
Consider a mini-batch of 2 pairs with cosine similarities (rows = queries, columns = listings, positives on the diagonal):
- Compute the loss for each row and the mean at \tau = 1, \tau = 0.1 and \tau = 0.02. What does each value say about the training signal?
- Explain mechanistically how temperature reshapes the gradient on negatives, and why a small \tau is more sensitive to false negatives. Add a duplicate of the positive listing as a third candidate for row 2 (score 0.8) and recompute row 2's loss at the three temperatures.
- What does a loss of \ln B mean, and what is the information- theoretic reason that more negatives help?
- Recommend a temperature strategy and two data-side fixes.
1. The numbers. Row 1 has logits (0.9, 0.6)/\tau with the positive first; row 2 has (0.7, 0.8)/\tau with the positive second. A 2-way softmax gives p_{\text{pos}} = \sigma(\Delta/\tau), where \Delta is the positive minus the negative score: \Delta_1 = 0.3, \Delta_2 = 0.1. The row loss is -\ln \sigma(\Delta/\tau).
| \tau | Row 1 | Row 2 | Mean |
|---|---|---|---|
| 1 | 0.554 | 0.644 | 0.599 |
| 0.1 | 0.049 | 0.313 | 0.181 |
| 0.02 | 0.000 | 0.007 | 0.003 |
(verified). At \tau = 1 the logits differ by at most 0.3, so the softmax is almost uniform: the loss cannot get far below the chance value \ln 2 = 0.693 however good the ranking is. This explains the plateau near 0.6 even though the ranking is already correct in both rows: the temperature is too high for cosine similarities, whose range is only [-1, 1]. At \tau = 0.02 the same margins become logit gaps of 15 and 5 and the loss is nearly zero: training "succeeds" almost immediately and then has little signal left except from the hardest examples.
2. Gradient reshaping and false negatives. The gradient of a row's loss with respect to its logits is p - \text{onehot} with p_j \propto \exp(s_{ij}/\tau), and with respect to the scores it is that divided by \tau. A large \tau gives every negative a similar share, so the push is spread over many easy negatives; a small \tau puts nearly all weight on the hardest negative(s) and ignores easy ones. That sharpens fine distinctions but means a single mislabelled hard negative owns the gradient. With the duplicate added (candidates 0.7, 0.8, 0.8 for row 2, positive second) the loss is 1.066 at \tau = 1, 0.862 at \tau = 0.1 and 0.697 at \tau = 0.02, compared with 0.644, 0.313 and 0.007 without the duplicate (verified). At \tau = 0.02 the loss rises a hundredfold and sits just above \ln 2: two candidates have identical scores so the model can never separate them, and the optimizer works on pushing apart two copies of the same product, distorting the space. That is the offline-recall drop the team sees, and the duplicate- heavy data is the cause.
3. \ln B and the number of negatives. With all scores equal, the softmax is uniform and the loss is \ln B (\ln 2 here). The InfoNCE bound is I(\text{query}; \text{listing}) \ge \ln B - \mathcal{L}, so a batch of B can certify at most \ln B nats of information, and more negatives both tighten the bound and make the discrimination task harder, which gives a richer gradient. Hence the push to large batches, memory queues (MoCo) or gradient caching.
4. Recommendation. Use a middle value, typically 0.05 to 0.1 for cosine-similarity encoders, or make \tau a learned parameter (CLIP initializes at 0.07 and clamps the scale at 100) and monitor it. Judge training by retrieval metrics on a held-out set, not the loss, since the loss value depends on \tau. Data-side fixes: deduplicate listings (or assign a product id and mask same-product pairs out of the negative set inside each batch), and be careful with mined hard negatives, dropping those whose scores are suspiciously close to the positive. Also consider a larger batch or a negative queue so that the hard negatives are informative rather than all noisy.
Share this question