Temperature and Sampling

From scores to choices

A language model does not output a single next token. It outputs scores, called logits, one per token in the vocabulary, representing how plausible each continuation is. Turning those scores into an actual choice is the job of the sampling step, and its settings change the model's behavior dramatically without changing the model itself.

The default conversion is the softmax function, which turns raw scores into probabilities that sum to one. But raw softmax on a 100,000-token vocabulary would give every token some tiny chance, including absurd ones. In practice the distribution is reshaped first, then sampled. Temperature, top-k, and top-p are the three controls for that reshaping.

Temperature

Temperature divides the logits before the softmax:

```

p_i = exp(z_i / T) / sum(exp(z_j / T))

```

When T is 1, the probabilities are whatever the model computed. As T approaches 0, the largest logit dominates completely: the model always picks the single most likely token. This is greedy decoding, fully deterministic. As T grows above 1, the differences between logits get compressed and the distribution flattens: unlikely tokens get chosen more often and output gets more varied, then more erratic.

In the OpenAI API, temperature runs from 0 to 2, with 1 as the default. The practical mapping most people use: near 0 for tasks with right answers, code, math, factual extraction, where you want the model's best guess every time. Around 0.7 to 1.0 for general writing, where some variety reads naturally. Above 1.0 for brainstorming and creative work, where surprise is the point and coherence can suffer.

Two things temperature does not do. It does not make the model smarter or dumber; it only changes how it samples from what it already computed. And it does not add knowledge. A model that does not know the answer will confabulate at any temperature. Lowering it just makes the confabulation more consistent.

Top-k sampling

Top-k restricts the choice to the k most likely tokens, discarding the rest before sampling. With k set to 40, the model picks among its top 40 candidates according to their renormalized probabilities. This cuts off the long tail of bizarre tokens while preserving variety among the plausible ones.

The weakness of top-k is that k is fixed while the shape of the distribution is not. Sometimes the model is confident and only three tokens deserve consideration; sometimes it is genuinely uncertain and a hundred are reasonable. A fixed k is too wide in the first case and too narrow in the second.

Top-p sampling

Top-p, also called nucleus sampling, adapts to the distribution. It sorts tokens by probability and keeps adding them to the candidate set until their cumulative probability reaches p, then samples from that set. With p at 0.9, the model considers however many tokens it takes to cover 90% of the probability mass: few when confident, many when uncertain.

This adaptivity is why top-p has largely replaced top-k as the default. A typical setting is p around 0.9 to 0.95. Setting p to 1.0 disables the filtering entirely. Most practitioners adjust either temperature or top-p, not both at once, since they interact: low temperature already concentrates the distribution, making top-p redundant.

Seeds and reproducibility

Sampling is random, so the same prompt can produce different outputs on different runs. Most APIs accept a seed value that makes the randomness deterministic: same prompt, same settings, same seed, same output. This is useful for testing and debugging.

The reproducibility has limits. It generally holds within one API version but can break when the provider updates the model or the serving infrastructure. Floating-point differences across hardware can also change which token wins a close call. Treat seeds as a debugging aid, not a guarantee.

Greedy decoding and beam search

Sampling is not the only way to choose tokens. Greedy decoding always picks the single most likely next token, which is what temperature 0 approximates. It is fast and deterministic but can paint the model into corners: the most likely first word sometimes leads to a worse overall sentence than a slightly less likely one would have.

Beam search hedges against this by keeping several candidate sequences alive at once. At each step it expands every candidate with its most likely continuations, then keeps only the top few by total score. A beam of width 5 explores five hypotheses in parallel and returns the best complete sequence. This was the standard decoding method for translation systems before sampling took over for open-ended generation. It produces higher-probability outputs than greedy decoding but costs more compute and tends toward safe, generic text, which is why chat models usually sample instead.

Choosing settings

The choice follows the task. Deterministic settings, temperature 0, for anything where consistency matters: classification, extraction, code generation, test suites. Moderate settings for open-ended writing where natural variation helps. High temperature only when you want genuine surprise and can tolerate incoherence.

One more consideration: evaluation. Benchmarks are usually run at temperature 0 so results are comparable across runs. A model that looks creative at temperature 1.2 and a model that looks precise at temperature 0 are the same model. When comparing outputs, compare them at the same settings.