Sampling and Temperature

At every step, a language model outputs a score for each token in its vocabulary. A softmax turns those scores into probabilities. Something then has to choose:

• Greedy: always take the most likely token. Deterministic, but tends to loop and repeat.
• Sampling: draw a token at random according to the probabilities. More natural, but occasionally picks something absurd from the long tail.

The knobs that shape the draw:

• Temperature divides the scores before the softmax. Below 1 sharpens the distribution toward the top choices; above 1 flattens it so unlikely tokens gain probability.
• Top-k keeps only the k most likely tokens and samples among them.
• Top-p (nucleus) keeps the smallest group of top tokens whose probabilities add up to p. When the model is confident, that group is small; when it is unsure, it widens.

So the same model can sound like a careful encyclopedia or a playful storyteller, depending only on how you sample.

The math

With logits $z_i$ and temperature $T$:

$$p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}$$

As $T \to 0$ this approaches greedy (argmax); as $T \to \infty$ it approaches uniform.