Block 04 / 07
The model outputs a probability distribution over tokens.
At each step the model assigns mass to every token, then you draw. GreedyGreedy decoding: picking the single highest-probability token at every step, with no search over alternative paths., temperatureA scalar that rescales a model’s logits before the softmax. Values below 1 sharpen the distribution toward the top tokens; values above 1 flatten it toward the tail., top-pNucleus sampling. Sorts tokens by probability and samples from the smallest set whose cumulative mass passes a threshold p, so the pool size adapts to the shape of the distribution instead of staying fixed., constrained decodingRestricting which tokens can be sampled at each step, with a grammar or schema, so the output is guaranteed to be well-formed.: those are policies over a distribution. There is no hidden true answer behind the softmaxThe function that turns a vector of logits into a probability distribution: exponentiate each score and divide by the sum, so the outputs are positive and add to 1.. Hallucination is mass on a fluent lie. Useful variation is mass you wanted. Calibration is whether “0.8” means 80% on a held-out set.
Accuracy is specified as a distribution, a decoding policy, and a verification procedure. TemperatureA scalar that rescales a model’s logits before the softmax. Values below 1 sharpen the distribution toward the top tokens; values above 1 flatten it toward the tail., schemas, and evaluations follow from that specification rather than from ad hoc preference.
Temperature 1.00 · calibrated, close to the training distribution
Drag the slider to change temperature. Toggle the cutoff to see which tokens survive nucleus sampling.
A decoding policy turns that distribution into one chosen token, repeated until the sequence ends. The three families below make the same choice, how much of the distribution’s tail to trust, in three different ways.
Deterministic
GreedyGreedy decoding: picking the single highest-probability token at every step, with no search over alternative paths. takes the single highest-probability token at every step. Beam search keeps the k best partial sequences instead of one and picks the best survivor at the end. Both are exact and repeatable, and both tend toward bland or looping text, since the safest token at each step compounds into a generic whole.
Stochastic, temperature-scaled
TemperatureA scalar that rescales a model’s logits before the softmax. Values below 1 sharpen the distribution toward the top tokens; values above 1 flatten it toward the tail. rescales the logitsA model’s raw, unnormalized scores for each candidate token, before the softmax turns them into a probability distribution. before the softmaxThe function that turns a vector of logits into a probability distribution: exponentiate each score and divide by the sum, so the outputs are positive and add to 1.: below 1 it sharpens the distribution toward the top tokens, above 1 it flattens it toward the tail. Top-k caps the pool to a fixed number of candidates before sampling, a blunt but cheap way to keep the draw away from the distribution’s long, unreliable tail.
Stochastic, adaptive
Sorts tokens by probability and keeps the smallest set whose cumulative mass passes a threshold p, then samples from that set. The pool grows or shrinks with the shape of the distribution itself: a handful of tokens when the model is confident, many more when it is spread out (Holtzman et al., 2019).
Paper
Nucleus sampling, and a clear argument that decoding is a policy. Pairs well with Guo et al. 2017 on calibration if you need the confidence side.
Book
Distributions, decision rules, why argmax is a policy. Older than LLMs. The math is the same.
Course
Prerequisite language for sampling: mass, expectation, and calibration.
The model assigns mass to tokens. Sampling, calibration, and constrained decoding are policies over that distribution.
Greedy, temperature, and nucleus sampling are decision rules on the same softmax. Variation in the output is a property of the policy, not a creativity setting with no mathematics.
A fluent falsehood is probability mass on a token sequence that is not a fact. The model did not “decide to lie”; it drew from a distribution that put mass there.
A schema or grammar restricts the token lattice so invalid JSON cannot be sampled. Extraction is decoding under a constraint, not a request written in the prompt.
Pick one extraction task and one open-ended task. Run the same prompts twenty times at temperature 0 and at 0.8 (and at one constrained-decoding setting if the API allows). Score schema validity, exact match or a factual check, and disagreement across runs. Plot the tradeoff.