Math foundations
Softmax: Turning Scores Into a Probability Distribution
The idea
A model scores three intents, or five product categories, or a thousand tokens, and spits out raw numbers called logits. Those are not probabilities — they can be negative and do not add up to anything. Softmax fixes both problems: it exponentiates each score so everything is positive, then divides by the total so the outputs form a clean probability distribution.
Softmax answers: how do I turn a vector of raw class scores into probabilities that are positive and sum to 100%?
Example: raw scores into a probability distribution
Three classes, three raw scores (logits). Softmax exponentiates and normalizes so the outputs are positive and sum to 100%. Temperature controls how sharp or hedged the result is.
Class A wins with 62.9%. Softmax exponentiates each score, so a small lead in raw logits becomes a larger lead in probability, and the three outputs always sum to 100%. At temperature 1 the raw score gaps map directly to probability gaps.
The math
Definition
Each output depends on every score, not just its own — raise one logit and the others' shares fall. Because it uses differences, adding a constant to all logits changes nothing.
Temperature
Dividing logits by T > 1 flattens the distribution (more hedged); T < 1 sharpens it toward the top class. With two classes, softmax reduces exactly to the sigmoid.
A simple application
Softmax is the output layer of nearly every multiclass classifier and language model. It generalizes the sigmoid from logistic regression to many classes, and it pairs with cross-entropy loss during training. Whether those probabilities are trustworthy is a separate question of calibration.