When you ask a language model to write an answer, it does not simply “know” the next word. Instead, it predicts many possible next tokens and assigns each one a probability.
For example, after the phrase:
The cat sat on the...
the model might predict:
Token → Probability
mat → 45%
floor → 25%
chair → 15%
roof → 8%
moon → 4%
pizza → 3%
Sampling settings like temperature, top-p, and top-k decide how the model chooses from these options.
Temperature: The Randomness Knob
Temperature controls how bold or conservative the model is when picking the next token.
A low temperature makes the model strongly favor the most likely tokens. This usually creates output that is focused, stable, and predictable.
temperature = 0.2
This is useful for tasks like:
- answering factual questions
- writing code
- summarizing documents
- following strict instructions
A high temperature gives less likely tokens a better chance. This can make the output more creative, varied, and surprising.
temperature = 1.2
This is useful for tasks like:
- brainstorming
- creative writing
- naming ideas
- exploring different tones
In simple terms:
Lower temperature = safer and more predictable Higher temperature = more creative and more random
Top-k: Choose From the Best k Options
Top-k limits the model to only the k most likely tokens.
If:
top_k = 3
and the top tokens are:
mat, floor, chair, roof, moon
the model can only choose from:
mat, floor, chair
Everything outside the top 3 is removed.
Top-k is easy to understand because it uses a fixed number. The downside is that it does not care how confident the model is. It always keeps the same number of choices, even when the model is very sure or very unsure.
Top-p: Choose From a Probability Pool
Top-p, also called nucleus sampling, works differently.
Instead of keeping a fixed number of tokens, it keeps the smallest set of tokens whose combined probability reaches a chosen value.
For example:
top_p = 0.9
means:
Keep adding the most likely tokens until their total probability reaches 90%.
If the model is very confident, top-p may keep only a few tokens. If the model is uncertain, it may keep many.
That makes top-p more flexible than top-k.
How They Work Together
These settings often work together.
temperature = 0.7
top_p = 0.9
top_k = 40
A simple way to think about it:
- Temperature reshapes the probabilities.
- Top-k removes everything except the best k options.
- Top-p keeps only the most useful probability mass.
- The model samples from what remains.
So temperature changes how adventurous the model feels, while top-p and top-k decide which options are even allowed.
Practical Settings
For factual or coding tasks:
temperature = 0.0 to 0.3
top_p = 0.8 to 1.0
For balanced assistant responses:
temperature = 0.5 to 0.8
top_p = 0.8 to 0.95
For creative writing:
temperature = 0.9 to 1.3
top_p = 0.9 to 1.0
There is no perfect setting for every task. The best values depend on what you want: accuracy, variety, creativity, or control.
Final Takeaway
Temperature, top-p, and top-k are all ways to control how a language model chooses its next token.
- Temperature controls randomness.
- Top-k limits choices to the best k tokens.
- Top-p limits choices to a flexible probability pool.
Use lower randomness when you want precision. Use higher randomness when you want creativity.
A good default for many tasks is:
temperature = 0.7
top_p = 0.9
top_k = 40
From there, adjust based on the kind of output you want.
Infographic Summary
Created by Vinay C
Originally published on LinkedIn.