Embeddings & Representation Learning
From word2vec to contrastive learning and two-tower models — how vectors are learned, compared and evaluated
Understand how embeddings are learned, compared and judged, and defend each choice in an interview: what an embedding is and the geometry of similarity (cosine vs dot product vs Euclidean distance, when they give the same ranking, normalization, anisotropy and hubness, both measured), classical word embeddings (skip-gram with negative sampling, the unigram^0.75 noise distribution, why SGNS is implicit factorization of a shifted PMI matrix, verified on a toy corpus; GloVe; item2vec and prod2vec for sessions), supervised embeddings (the penultimate layer, sampled and hierarchical softmax for million-class outputs, embedding tables and the hashing trick with collision arithmetic), contrastive learning (InfoNCE with temperature, in-batch negatives, hard and false negatives, batch-size dependence, SimCLR, CLIP and zero-shot classification, SBERT, triplet and margin losses, and why collapse needs negatives, stop-gradient or variance regularizers, with a measured collapse experiment), two-tower retrieval models (dot-product scoring, ANN serving, the log-Q sampling-bias correction verified by simulation, why cross features cannot enter the towers, content features for cold start), self-supervised pretext tasks (MLM, next-token, masked autoencoders) and the pretrain-finetune paradigm, multilingual and multimodal spaces with the modality gap, evaluation (neighbour sanity checks, recall@k, MRR, NDCG, linear probes, MTEB-style benchmarks, the offline-online gap), operating embeddings (dimension and storage arithmetic, Matryoshka embeddings with a measured experiment, versioning and why vectors from two model versions are not comparable, drift), and a worked example with hand-computed InfoNCE at two temperatures, a log-Q corrected two-tower score and recall/NDCG on a ranked list.
Practice questions (6)
-
View →
Explain Negative Sampling for a 2M-Item item2vec Model
Intermediate · Free -
View →
Diagnose a Bi-Encoder Fine-Tune: What Temperature Does to InfoNCE
Intermediate · Free -
View →
Design a Two-Tower Retriever With Cold-Start Handling
Advanced -
View →
Explain the Log-Q Correction to a Teammate Who Wants to Remove It
Advanced -
View →
Evaluation Plan for Swapping the Embedding Model of a RAG System
Advanced