LightGBM

LightGBM is a gradient boosting framework that builds an ensemble of decision trees sequentially — each tree corrects the mistakes of all previous trees. It consistently ranks among the top-performing algorithms on tabular data in Kaggle competitions and production ML systems. Understanding its internals and hyperparameters is essential for any ML practitioner.

What Is Gradient Boosting?

Gradient boosting builds a model F(x) as a sum of M trees:

Where:

• F_0(x) is the initial prediction (usually the mean of the target)
• h_m(x) is the m-th tree
• \eta (eta) is the learning rate

Each tree h_m is trained not on the raw target labels, but on the negative gradient of the loss function — the residual errors the previous ensemble failed to explain.

Iteration 1: F_1(x) = F_0(x) + η × Tree_1(residuals_0)
Iteration 2: F_2(x) = F_1(x) + η × Tree_2(residuals_1)
Iteration 3: F_3(x) = F_2(x) + η × Tree_3(residuals_2)
...
Iteration M: F_M(x) = F_{M-1}(x) + η × Tree_M(residuals_{M-1})

Intuition: Imagine you're trying to hit a target with a bow. You fire an arrow (Tree 1), see where it lands, then fire another arrow correcting for the previous miss (Tree 2), and so on. Each arrow corrects the accumulated error of all prior arrows. The learning rate controls how much you adjust each shot.

LightGBM: Leaf-Wise vs Level-Wise Tree Growth

Most gradient boosting implementations (e.g., older XGBoost) grow trees level-wise (breadth-first): all nodes at depth d are split before moving to depth d+1. This produces symmetric, balanced trees.

LightGBM uses leaf-wise growth: at each step, find the single leaf with the highest potential gain across the entire tree and split it — regardless of level.

Level-wise growth (XGBoost default):        Leaf-wise growth (LightGBM default):
After 4 splits:                             After 4 splits:

         [Root]                                     [Root]
        /      \                                   /      \
      [L1]    [R1]           →              [L1]         [R1]
      /  \    /  \                          /  \           \
   [LL] [LR][RL] [RR]                   [LL] [LR]         [RR]
                                         /  \
                                       [LLL][LLR]

Balanced: 4 leaves, depth 2               Unbalanced: 5 leaves, deeper on left

Gain per split: average                   Gain per split: always maximum

Why leaf-wise is better: By always splitting the most informative leaf, LightGBM achieves lower training loss with fewer trees and fewer total leaf nodes. This translates to faster training and often better accuracy for a fixed number of leaves.

The risk: Leaf-wise growth can overfit more easily on small datasets because it grows very deep asymmetric trees. This is why num_leaves is the primary complexity control in LightGBM (not max_depth).