Math foundations

Entropy and Information Gain: How a Split Reduces Uncertainty

The binary entropy curve peaking at a fifty-fifty class mix

The idea

A pile of rows that is 50% fraud and 50% legit is maximally uncertain — a coin flip. A pile that is 99% legit is nearly settled. Entropy puts a number (in bits) on that uncertainty. Information gain measures how much a yes/no split shrinks it, and that is exactly how a decision tree decides which question to ask first.

Entropy answers: how uncertain is this label mix? Information gain answers: how much did this split clean it up?

Example: how much a split reduces uncertainty

The curve is entropy for a two-class node: highest at a 50/50 mix, zero when pure. Set the parent mix, then split the rows into two children. Information gain is the drop in weighted entropy.

Parent positives: 50%

Rows in left child: 50%

Left child positives: 85%

Parent entropy

1.00

Weighted children

0.61

Right child is 15% positive.

Information gain

0.39

The parent node carries 1.00 bits of uncertainty. After the split the weighted child entropy drops, leaving 0.39 bits of information gain — a useful split. A tree greedily picks the feature with the highest gain.

The math

Entropy (bits)

H(S) = − Σ pᵢ · log₂(pᵢ)

Sum over each class probability pᵢ. For two classes, entropy peaks at 1 bit when the split is 50/50 and falls to 0 when the node is pure. Rare classes contribute little.

Information gain

Gain = H(parent) − Σ (n_child / n) · H(child)

Subtract the size-weighted entropy of the children from the parent's entropy. A tree evaluates this for every candidate feature and greedily picks the highest gain.

A simple application

Information gain is the splitting criterion behind decision trees (Gini impurity is a close cousin). The same entropy term is the cross-entropy loss classifiers minimize, and it connects to maximum likelihood: minimizing cross-entropy is maximizing the likelihood of the labels.