Clustering & Dimensionality Reduction
Most of the data a data scientist meets has no labels. Nobody has tagged which customers "belong together", which server logs are anomalous, or which of 768 embedding dimensions actually carry signal. Unsupervised learning is the toolkit for that situation: it finds structure in X alone, with no y to check against. That last clause is what makes it hard — there is no loss on a held-out set that tells you whether your clusters are "right", so you have to understand the assumptions each algorithm makes and evaluate results with judgement rather than a single number.
In interviews, clustering and dimensionality reduction questions separate people who have run KMeans(n_clusters=5).fit(X) from people who can say why k-means fails on the given data, why the t-SNE plot's cluster sizes mean nothing, or how PCA is derived from the covariance matrix. Expect questions like "how would you choose k?", "when would you prefer DBSCAN?", "why must you standardise before PCA?", and "walk me through PCA on this 2-D dataset". Every one of those is covered here, with numbers.
The subject splits into two halves that are more related than they look. Clustering groups rows (which observations are similar?); dimensionality reduction compresses columns (which directions of variation matter?). In practice you often chain them: reduce with PCA, then cluster in the reduced space, then visualise with UMAP. Feature scaling — which both halves depend on — is covered in depth in the Feature Engineering subject; here we only state that it matters and why.
Goals of unsupervised learning
Supervised learning has an unambiguous target: minimise a loss between predictions and labels. Unsupervised learning has several distinct goals, and it pays to name which one you are pursuing before choosing an algorithm:
| Goal | Question it answers | Typical tools |
|---|---|---|
| Clustering | Which observations naturally group together? | k-means, hierarchical, DBSCAN, GMM |
| Dimensionality reduction | Which few directions explain most of the variation? | PCA, SVD, autoencoders |
| Visualisation | How can I look at 100-dimensional data? | PCA (2 components), t-SNE, UMAP |
| Density estimation / anomaly detection | Which points are unlike the rest? | GMM, DBSCAN noise labels, isolation forest |
| Representation learning | What compact features can feed a downstream model? | PCA, embeddings, autoencoders |
Two facts follow from "no labels":
- The algorithm cannot tell you if it is wrong. k-means will happily return 5 clusters on uniformly random data. Every result needs a sanity check — a plot, a domain expert, or downstream task performance.
- Preprocessing choices are modelling choices. Distance-based methods (k-means, DBSCAN, hierarchical) treat a feature measured in dollars and a feature measured in years as equally important per unit. Standardising first is not optional hygiene; it changes the answer.