Temperature vs top-p differ in what they touch inside a model’s output. Temperature rescales every logit before softmax runs, so it reshapes the whole probability distribution. Below T = 1 it sharpens the leading token’s share. Above T = 1 it flattens the distribution toward the tail. Every token keeps a non-zero probability at any temperature; nothing gets removed. Top-p, also called nucleus sampling, works differently. It sorts tokens by probability, keeps the smallest set whose cumulative mass reaches p, then discards everything else outright. The survivors then get renormalised, and the model samples from that smaller set. So temperature reweights the full distribution, while top-p truncates it and deletes the tail.
Every large language model (LLM) exposes two sampling knobs through its application programming interface (API): temperature and top-p. Most developers nudge both until the output looks right, then move on. That habit works, until it doesn’t, and the failure is hard to debug without knowing the mechanism underneath.
Instead, this guide shows the actual arithmetic, not a vague description. The same four candidate tokens run through both settings, with every probability worked out to three decimal places. So you can see exactly which number changes, and why.
Sampling settings are only one lever, though. Other choices change model behaviour at inference time too, the comparison drawn in quantization vs distillation. Temperature and top-p, by contrast, act on the probability distribution itself, one step before a token gets chosen.
How a Model Picks the Next Token
A model scores every token in its vocabulary at each step. Those raw scores are called logits, one number per token, and they can be positive, negative, or zero.
Softmax then turns logits into probabilities. It exponentiates each logit, then divides by the sum of all the exponentials, so the results add up to 1. That layer shows up across deep learning generally, not only in language models, the same foundation covered in machine learning vs deep learning.
The decoder then picks a token one of two ways. Greedy decoding always takes the single highest-probability token, the same choice every time given the same input. Sampling instead draws a token according to the probability distribution, so a lower-probability token can still get picked. Temperature and top-p both shape that sampling step, not the greedy one.
What Temperature Does
Temperature rescales the logits before softmax runs. The formula is:
pi = exp(zi / T) / Σ exp(zj / T)
Here zi is the logit for token i, and T is the temperature. Dividing every logit by T changes how sharply softmax separates the leader from the rest.
A temperature below 1 sharpens the distribution: the leading token gains probability, and the tail loses it. A temperature above 1 flattens it instead; probability moves from the leader toward the tail. So as T approaches 0, the distribution converges on greedy decoding, always picking the top token. Division by zero is undefined in the formula itself, so most APIs special-case T = 0 to mean exactly that: always take the top token.
One property matters more than the rest. Temperature never removes a token. Every token keeps a non-zero probability, however small, at any temperature setting. That is the property separating it from top-p.
Advantages of temperature.
- Simple, single-number control over how predictable sampling is.
- Keeps every token reachable, so rare but valid continuations stay possible.
- Applies uniformly at every step, with no dependence on how confident that step is.
Disadvantages of temperature.
- A high setting can pull real probability into tokens that make little sense.
- It does not adapt to how peaked or flat the distribution already is.
- Tuning it well often takes trial and error, not a fixed rule.
What Top-p Does
Top-p is also called nucleus sampling, a method from Holtzman et al. (2019). It works on probabilities, after softmax has already run, not on the raw logits.
The steps stay fixed. First, sort every token by probability, highest first. Then keep the smallest set of tokens whose cumulative probability reaches p. Discard everything outside that set entirely, then renormalise what remains so the kept probabilities sum to 1 again.
It then samples from that smaller, renormalised set. The size of the nucleus is adaptive, and that is the core idea behind it. For example, on a confident step, one or two tokens might already cover p. On an uncertain step, the nucleus can hold dozens of tokens instead.
Advantages of top-p.
- Adapts automatically to how confident the model is at each step.
- Removes implausible tail tokens outright, instead of merely shrinking their share.
- Behaves consistently across vocabularies of very different sizes.
Disadvantages of top-p.
- A low p value can truncate the nucleus to just one or two tokens.
- Discarded tokens are gone for that step, with no partial credit.
- Choosing p still takes trial and error, much like temperature does.
Temperature vs Top-p: Comparison Table
| Aspect | Temperature | Top-p |
|---|---|---|
| What it operates on | Logits, before softmax | Probabilities, after softmax |
| What it changes | Shape of the whole distribution | Which tokens stay eligible |
| Are tokens removed | No, never | Yes, the tail is discarded |
| Effect on the tail | Shrinks or grows, never zero | Cut off entirely past the cutoff |
| Typical range | 0 to 2, in most APIs | 0 to 1, by definition |
| Adaptivity per step | Fixed rescaling, same at every step | Nucleus size adapts to confidence |
| Effect at the low extreme | Converges on greedy decoding | Nucleus can shrink to one token |
| Effect at the high extreme | Distribution flattens toward uniform | Nucleus approaches the full vocabulary |
| Order of application | Applied first, to the logits | Applied after temperature, to probabilities |
| Relationship to greedy decoding | T = 0 is special-cased as greedy | Not directly tied to greedy decoding |
| Relationship to top-k | Independent of top-k entirely | An adaptive alternative to top-k’s fixed count |
| Good for | Controlling overall predictability | Trimming implausible tail tokens |
| Main failure mode | Too high turns output incoherent | Too low removes useful variety |
| Tune alongside the other setting | Usually adjust one, not both | Same guidance applies here |
Worked Example: The Same Four Tokens
Four candidate tokens carry these logits: cat 2.0, dog 1.0, bird 0.5, fish 0.1. The same four tokens run through temperature first, then top-p.
| Token | T = 0.5 | T = 1.0 | T = 2.0 |
|---|---|---|---|
| cat | 0.828 | 0.575 | 0.406 |
| dog | 0.112 | 0.211 | 0.246 |
| bird | 0.041 | 0.128 | 0.192 |
| fish | 0.019 | 0.086 | 0.157 |
At T = 0.5, cat takes 82.8% of the mass, and fish nearly vanishes at 1.9%. At T = 2.0, cat drops to 40.6%, while fish climbs to 15.7%. Every token still holds a non-zero probability at every temperature in this table.
Now apply top-p = 0.9 at T = 1.0, using the middle column above.
| Token | Probability | Cumulative |
|---|---|---|
| cat | 0.575 | 0.575 |
| dog | 0.211 | 0.786 |
| bird | 0.128 | 0.914 ← threshold reached |
| fish | 0.086 | 1.000 — discarded |
The nucleus is {cat, dog, bird}. Fish is removed entirely, then the three survivors get renormalised and sampled from. The cutoff lands at 0.914, slightly above 0.9, because the rule keeps the smallest set that reaches the threshold, not the largest set that stays under it.
Same four tokens, two different knobs, two different kinds of change. Temperature moved every bar’s height. Top-p deleted one bar completely and rescaled the rest.
The Order They Are Applied
Temperature and top-p do not run in parallel. Temperature applies first, to the raw logits. Top-p, or top-k, applies second, to the probabilities that come out of that rescaled softmax.
The kept set then gets renormalised, and the model samples from it. So changing temperature changes which tokens even reach the top-p cutoff. A higher temperature flattens the distribution first, which can pull more tokens into the nucleus at the same p value.
That dependency is exactly why tuning both settings at once gets confusing. Move one, and the other’s effective behaviour shifts too, even though its own value never changed. Many API docs recommend adjusting one of the two, not both together.
Where Top-k Fits
Top-k is the third common knob, and it works on the sorted probability list too. It keeps a fixed number of top tokens, k of them, regardless of how confident the model is at that step.
Top-p keeps a variable number instead, one that adapts to the shape of the distribution. So a confident step might need only one or two tokens to reach p. Instead, an uncertain step might need dozens.
That adaptivity is nucleus sampling’s main argument over top-k. A fixed k can feel too generous on a confident step, or too strict on an uncertain one. Top-p adjusts on its own, without a separate setting for every situation.
Which One to Change
Start with one setting, not two. Most API docs recommend leaving the other at its default while you adjust the first, since together the two interact in ways that are hard to reason about.
Temperature is the more direct lever for overall unpredictability, since it reshapes the entire distribution in one pass. Top-p is the more direct lever for trimming implausible options, since it removes them outright rather than merely shrinking their share.
Still, neither setting touches the model’s weights. Changing the weights instead, through LoRA, QLoRA, or full fine-tuning, changes what the logits look like before sampling even starts. Retrieval is a third option again: pulling in fresh context through retrieval-augmented generation instead of fine-tuning changes what the model has available to score, not how it samples from the scores. Sampling settings, weight changes, and retrieval each shift output for a different reason, and mixing them up wastes debugging time.
Interview Questions
Frequently Asked Questions
Wrapping Up
Temperature vs top-p comes down to reshaping versus removing. Temperature rescales logits before softmax, sharpening or flattening the distribution while every token stays reachable. Top-p truncates the resulting probabilities instead, discarding the tail and renormalising what is left.
Keep the worked example close. At T = 1.0, top-p = 0.9 kept {cat, dog, bird} and discarded fish, with the cumulative sum landing at 0.914. Change the temperature first, and that same cutoff can keep or drop a different set of tokens entirely.
So start by adjusting one setting, not both. Most API documentation says exactly that, and the worked example shows why: temperature changes the inputs that top-p’s cutoff depends on.
Related reading on DiffStudy:
- Quantization vs Distillation
- Machine Learning vs Deep Learning
- LoRA vs QLoRA vs Full Fine-Tuning
- Retrieval-Augmented Generation vs Fine-Tuning
- CS Fundamentals hub
