Practice — Embeddings & Representation Learning (6 questions)

A shopping app wants "similar items" for a catalogue of 2,000,000 products. An engineer treats each of 50M browsing sessions as a sentence of item ids and trains skip-gram with a full softmax over all items, embedding size 128. One epoch is projected to take weeks. A colleague says "just use negative sampling with k = 10 negatives" but cannot explain what it optimizes, and another asks why the noise words are drawn from "frequency to the power 0.75" instead of uniformly or by raw frequency.

1. Compare the output-layer cost per (centre, context) pair of the full softmax and of negative sampling with k = 10.
2. Write the negative-sampling loss for one pair and compute it for an observed pair with score 2.0 and two sampled negatives with scores 0.5 and -1.0. Which term dominates and why?
3. Explain the unigram^0.75 noise distribution with a three-item example whose frequencies are 0.9, 0.09 and 0.01.
4. State what negative sampling implicitly optimizes at its optimum and what changing k does to it.
5. Name three practical adjustments for session data (not for text) and the cold-start limitation that remains.

You fine-tune a product-search bi-encoder (query tower and listing tower) with in-batch InfoNCE on (query, clicked listing) pairs. With temperature \tau = 1 the training loss plateaus near 0.6 and recall@10 barely moves. Lowering to \tau = 0.02 makes the loss go almost to zero within a few hundred steps, but offline recall on a held-out set gets worse and the team notices the data contains many duplicated listings (the same product from several sellers).

Consider a mini-batch of 2 pairs with cosine similarities (rows = queries, columns = listings, positives on the diagonal):

1. Compute the loss for each row and the mean at \tau = 1, \tau = 0.1 and \tau = 0.02. What does each value say about the training signal?
2. Explain mechanistically how temperature reshapes the gradient on negatives, and why a small \tau is more sensitive to false negatives. Add a duplicate of the positive listing as a third candidate for row 2 (score 0.8) and recompute row 2's loss at the three temperatures.
3. What does a loss of \ln B mean, and what is the information- theoretic reason that more negatives help?
4. Recommend a temperature strategy and two data-side fixes.