Logistic Regression & Linear Classifiers
Logistic regression is the first classifier every data scientist should reach for and the last one most should give up. It is fast, convex, interpretable, produces genuinely calibrated probabilities, scales to hundreds of millions of sparse features, and is the model behind an astonishing share of production click, fraud, churn and credit systems. It is also the standard interview probe for whether a candidate understands the difference between a score, a probability and a decision.
This subject builds the model from the question "why not just run linear regression on a 0/1 target?", derives the log-loss objective from maximum likelihood, shows why the decision boundary is a hyperplane, and then spends most of its time on the parts practitioners get wrong: reading coefficients as odds ratios, choosing a threshold that reflects real costs instead of defaulting to 0.5, checking calibration, handling class imbalance without destroying the probability scale, and knowing when a linear SVM, naive Bayes or a tree ensemble is the better tool.
Two boundaries with sibling subjects in this track. The full treatment of ROC curves, precision-recall curves and AUC lives in the Model Evaluation Metrics subject; here we use them only as far as threshold choice requires. Penalising the coefficients (L1/L2, the C parameter) is the Regularization (L1/L2) subject; here we only explain why logistic regression needs it.
Why Not Linear Regression on a 0/1 Target?
Fitting y \in \{0, 1\} with OLS gives the linear probability model \hat p = \mathbf x^\top\boldsymbol\beta. It is not useless — economists use it for average effects — but as a classifier it has three structural defects:
- Predictions escape [0, 1]. A customer with large feature values gets \hat p = 1.3 or -0.2. There is no probability interpretation.
- Constant marginal effects are wrong. OLS says one more unit of x adds \beta to the probability everywhere. Real probabilities saturate: going from very unlikely to slightly less unlikely is a small change, going from 40% to 60% is a big one.
- Heteroscedasticity by construction. \text{Var}(y \mid \mathbf x) = p(1-p), which varies with \mathbf x, so OLS standard errors are wrong and OLS is not efficient (see the Linear Regression subject).
Logistic regression fixes all three by modelling not p but a transformation of p that lives on the whole real line.