Upgrading to a Bigger Embedding Made Search Worse
Your k-NN recommender uses 2-dimensional item embeddings and works well: for a typical query point, the nearest neighbor is clearly much closer than the farthest point in the set. You swap in a fancier 1,000-dimensional embedding model, expecting sharper, more meaningful neighbors. Instead, recommendations get noticeably worse — nearest and farthest neighbors start looking almost interchangeable.
As the number of (roughly independent, noise-like) dimensions grows very large, what happens to the ratio of the farthest point's distance to the nearest point's distance, for a fixed query?
It approaches 1 — nearest and farthest neighbors become almost the same distance away.
This is the curse of dimensionality's effect on distance metrics, sometimes called distance concentration. With many roughly independent coordinates, a point's squared Euclidean distance to any other point is a sum of many independent per-coordinate contributions. By a law-of-large-numbers argument, that sum concentrates tightly around its mean as dimensionality grows — the relative spread (standard deviation divided by mean) of distances shrinks toward zero, even though the absolute distances keep growing. The practical consequence: in very high dimensions, essentially every point ends up at nearly the same distance from your query point, so "nearest neighbor" stops being a meaningful, well-separated concept — you're no longer finding the closest item, you're picking almost arbitrarily among a crowd of nearly-tied candidates.
This is exactly backwards from the intuition that more dimensions means more room to separate things — more independent, noise-like dimensions means less relative contrast, not more. It only bites when dimensions genuinely add spread-out, low-structure information; a 1,000-dim embedding from a model that concentrates real signal into a much lower effective (intrinsic) dimensionality can still work fine, which is precisely why the fix is not "always use fewer raw dimensions" but: use a similarity metric less sensitive to this (cosine similarity over normalized vectors is far more common than raw Euclidean distance for exactly this reason), and/or project onto the embedding's actual effective dimensionality (PCA, or a purpose-trained lower-dim model) rather than trusting Euclidean distance on hundreds of raw dimensions with mixed signal-to-noise.
Share this question