Temperature, Top-p, and How AI Chooses Its Next Word

A language model does not "write" a sentence. At every step it produces a probability distribution over its entire vocabulary — tens of thousands of possible next tokens, each with a score — and then something has to pick one. Temperature and top-p are the knobs that control that pick. They don't change what the model knows; they change how boldly it gambles on the less-likely options it's already considering.

That's the whole idea, and it's worth saying up front because the settings are usually explained backwards. Temperature is not a "creativity dial" in any deep sense, and top-p is not "quality." Both are just different rules for sampling from a list of candidates the model already ranked. Once you see the distribution underneath, every piece of advice about these settings stops being folklore and starts being obvious. This post is about that distribution — what it is, what the knobs do to it, and how to set them without cargo-culting.

Table of contents

• Key takeaways
• The one diagram in your head: logits → probabilities → a pick
• The softmax, with actual numbers
• Temperature, derived: dividing the logits
• What temperature actually does
• Top-p and top-k: truncating before you gamble
• The full sampling zoo: top-k, top-p, min-p, typical
• Repetition, frequency, and presence penalties
• Beam search, and why chat models abandoned it
• A quick comparison
• When to turn it down, when to turn it up
• Settings per task: a concrete cheat sheet
• Sampling and reasoning models
• Why temperature 0 isn't fully deterministic
• Seeds, reproducibility, and what you can actually promise
• Common mistakes
• FAQ

Key takeaways

• Every token is a sample from a probability distribution. The model outputs a score for every word in its vocabulary; a sampling step turns those scores into one chosen token. Temperature and top-p are rules for that step.
• Temperature reshapes the distribution. Low temperature sharpens it (the top choice dominates); high temperature flattens it (long-shots get a real chance). Temperature 0 means "always take the single most likely token."
• Top-p (nucleus sampling) truncates the distribution. It keeps only the smallest set of top tokens whose probabilities add up to p, then samples from that set. It adapts to how confident the model is.
• Top-k is the blunt cousin of top-p: keep the k most likely tokens, fixed count, then sample. Simpler, less adaptive.
• Low = faithful and repetitive; high = varied and risky. There's no universally "correct" value — it depends on whether you want the safe answer or the surprising one.
• Temperature 0 is not truly deterministic in most real systems. Floating-point math, batching, and hardware ordering can still flip a token. "Greedy" reduces variance; it rarely eliminates it.
• These knobs don't fix a bad prompt. If the model doesn't know something, no temperature makes it know. Sampling only chooses among what the model already thinks is plausible.

The one diagram in your head: logits → probabilities → a pick

Here's what happens each time the model emits a token, roughly. If you want the fuller picture of how the surrounding system works, how AI chatbots work covers the rest of the loop; this section zooms into the final step.

1. Logits. The model computes a raw score — a logit — for every token in its vocabulary. These are unbounded numbers: some positive, some negative, no particular scale.
2. Softmax → probabilities. A function called softmax squashes those logits into a probability distribution: every value between 0 and 1, all of them summing to 1. Now "the" might be 0.18, "a" 0.09, "quantum" 0.0001, and so on across the whole vocabulary.
3. Sampling. A selection rule picks one token from that distribution. Append it to the text, feed everything back in, and repeat for the next token.

Temperature acts in step 2 — it changes the shape of the probabilities before they're formed. Top-p and top-k act in step 3 — they change which candidates are eligible to be sampled. That's the entire mechanism. Everything else is tuning.

The key realization: the model's "knowledge" lives in the logits. It has already decided that "the" is more likely than "quantum" here. Sampling can only redistribute chances among the options the model surfaced — it cannot invent a good token that scored near zero, and it cannot make a wrong token become right.

The softmax, with actual numbers

Everything about temperature and top-p is downstream of one function, so it pays to see it work once with real arithmetic rather than hand-waving. Softmax takes a list of logits and turns them into probabilities. For a logit z_i, the probability is:

P(i) = exp(z_i) / Σ_j exp(z_j)

In words: exponentiate every logit, then divide each by the total. The exponential is the important part. It is monotonic (bigger logit → bigger probability, always) but it is also convex, which means it stretches large values apart much faster than small ones. A gap of 2 between two logits does not become a gap of 2 in probability space; it becomes a ratio of exp(2) ≈ 7.4×.

Take a toy vocabulary of four tokens with these logits after the model reads "The cat sat on the":

Token	Logit
mat	3.0
floor	2.0
roof	1.0
bicycle	0.0

Exponentiate: exp(3)=20.09, exp(2)=7.39, exp(1)=2.72, exp(0)=1.00. The sum is 31.20. Divide each through:

Token	exp(logit)	Probability
mat	20.09	0.644
floor	7.39	0.237
roof	2.72	0.087
bicycle	1.00	0.032

So "mat" is not just the top pick — it holds nearly two-thirds of the probability mass, even though its logit lead over "floor" was only 1.0. That is the exponential at work: modest differences in logits become lopsided differences in probability. Hold this table in your head, because temperature is nothing more than a way to squeeze or stretch those four logits before they hit the exponential.

Two properties worth noting. First, softmax is shift-invariant: adding the same constant to every logit changes nothing, because the constant factors out of the ratio. Only the gaps between logits matter, never their absolute size. Second, no probability is ever exactly zero — even "bicycle" keeps a 3.2% share here. Every token in the vocabulary always retains a sliver of the mass. That is precisely why truncation methods like top-p and top-k exist: to amputate that long, non-zero tail before it can bite.

Temperature, derived: dividing the logits

Now insert temperature. The temperatured softmax is:

P(i) = exp(z_i / T) / Σ_j exp(z_j / T)

The only change is z_i / T. Every logit is divided by the temperature T before exponentiation. Run our four tokens through three settings.

T = 1 (leave logits alone) reproduces the table above: mat 0.644, floor 0.237, roof 0.087, bicycle 0.032.

T = 0.5 (divide logits by 0.5, i.e. double them). New logits: 6.0, 4.0, 2.0, 0.0. Exponentiate and normalize:

Token	Probability at T=0.5
mat	0.867
floor	0.117
roof	0.016
bicycle	0.002

Halving the temperature pushed "mat" from 0.64 to 0.87. The distribution sharpened; the leader pulled away.

T = 2 (divide logits by 2). New logits: 1.5, 1.0, 0.5, 0.0. Normalize:

Token	Probability at T=2
mat	0.428
floor	0.260
roof	0.158
bicycle	0.096

Doubling the temperature dropped "mat" from 0.64 to 0.43 and lifted "bicycle" from 3% to nearly 10%. The distribution flattened; the long shots got real odds.

Two edge cases fall straight out of the formula. As T → 0, dividing by a vanishing number blows the largest logit up relative to the rest, so its probability approaches 1 and everything else approaches 0 — that is greedy decoding, and it is why "temperature 0" is shorthand for "always take the argmax" even though you cannot literally divide by zero (implementations special-case it). As T → ∞, every z_i / T approaches 0, every exp(0) = 1, and the distribution becomes uniform — pure noise, every token equally likely. Real temperature knobs live on the interesting stretch in between, usually 0 to 2.

Notice what temperature does not do: it never reorders the tokens. "mat" is the most likely token at T=0.5, T=1, and T=2. Temperature only changes how much the leader leads. That is the single most useful thing to remember when someone claims a higher temperature made the model "think differently" — it did not change the ranking, only the confidence.

What temperature actually does

Temperature is a single number you divide the logits by before softmax. That sounds trivial; the effect is not.

• Divide by a small number (temperature below 1) and you amplify the gaps between logits. Big scores get relatively bigger, small ones relatively smaller. The distribution gets sharp — a peak. The top token increasingly dominates. At the limit, temperature approaches 0 and the peak becomes a spike: the single highest-scoring token gets essentially all the probability. This is called greedy decoding.
• Divide by a large number (temperature above 1) and you compress the gaps. Scores move toward each other, the distribution gets flat, and unlikely tokens get a meaningfully larger share. Set it high enough and you approach uniform randomness — the model starts picking near-gibberish because everything is roughly equally likely.
• Temperature = 1 leaves the logits untouched: you sample from the model's "native" distribution, exactly as trained.

A useful mental image: temperature is a contrast dial on the probability distribution. Turn contrast up (low temperature) and the brightest option washes out everything else. Turn contrast down (high temperature) and the dim options become visible and start getting chosen.

This is why "raise the temperature for more creativity" is only half true. You're not adding ideas. You're raising the odds that the model commits to a less-probable continuation it was already entertaining. Sometimes that less-probable path is a fresh metaphor. Sometimes it's a factual error or a broken sentence. High temperature buys variance, and variance cuts both ways.

Top-p and top-k: truncating before you gamble

Temperature reshapes the whole distribution, including a very long tail of tokens that are individually near-impossible but collectively numerous. At high temperature those tail tokens can accumulate enough probability to occasionally get picked — which is how you get sudden nonsense words. Top-p and top-k exist to cut off that tail before sampling.

Top-k is the simple version: keep only the k highest-scoring tokens, throw away the rest, renormalize, and sample. If k = 40, only the 40 best candidates are ever eligible. The weakness is that k is fixed regardless of context. Sometimes 40 candidates is far too many (the model is very sure of the next token and only 2 are reasonable); sometimes it's too few (a genuinely open choice with 300 valid continuations).

Top-p, also called nucleus sampling, fixes that by being adaptive. Instead of a fixed count, you set a probability mass. With p = 0.9, you keep the smallest set of top tokens whose probabilities add up to at least 0.9, discard the rest, and sample from that "nucleus."

• When the model is confident — one token already holds 0.92 of the mass — top-p keeps almost nothing else. The pick is nearly forced. Good: you don't want randomness where the answer is obvious.
• When the model is uncertain — the top token is only 0.15 and probability is spread across dozens of options — top-p keeps a wide set. Good: it allows variety exactly where variety is legitimate.

That adaptivity is why top-p is the more popular default. It self-adjusts to the model's confidence instead of imposing a fixed cutoff.

You can use temperature and top-p together, and most systems do. A common recipe is a moderate temperature to set overall boldness, plus a top-p around 0.9–0.95 as a guardrail that clips the worst of the tail. They're not competitors; they operate at different stages.

The full sampling zoo: top-k, top-p, min-p, typical

Top-p and top-k are the two truncation methods you will meet everywhere, but they are not the only ones, and the newer variants exist precisely because top-p has a known failure mode. It is worth seeing the family together, because each one answers a slightly different question about which tokens deserve to stay in the running.

Return to the four-token example at T=1: mat 0.644, floor 0.237, roof 0.087, bicycle 0.032.

• Top-k (k = 2) keeps a fixed count: the two highest, mat and floor. Renormalize over 0.644 + 0.237 = 0.881, giving mat 0.731 and floor 0.269. Roof and bicycle are gone regardless of how much mass they held. The flaw is rigidity: k is the same whether the model is certain or torn.
• Top-p (p = 0.9) keeps the smallest set summing to 0.9. Mat alone is 0.644 — not enough. Add floor: 0.881 — still short of 0.9. Add roof: 0.968 — over the line. So the nucleus is {mat, floor, roof}, and bicycle is dropped. Notice the set size emerged from the data rather than being fixed in advance. That is the whole appeal.
• Min-p (min_p = 0.1) takes a different tack: keep every token whose probability is at least min_p × (probability of the top token). Here the threshold is 0.1 × 0.644 = 0.0644. Mat (0.644), floor (0.237), and roof (0.087) all clear it; bicycle (0.032) does not. Min-p scales its cutoff to the model's confidence: when the top token is dominant the bar is high and the pool shrinks; when the field is even the bar is low and more tokens survive. Many practitioners find it more robust than top-p at high temperatures, because it will not admit a token just because a long tail of mediocre options collectively crossed a mass threshold.
• Typical sampling is the most exotic of the common four. Instead of keeping the most probable tokens, it keeps tokens whose surprise (negative log-probability) is closest to the distribution's average surprise — the entropy. The intuition from information theory is that natural language tends to sit near its expected information content, so both the boringly obvious token and the wildly improbable one can be atypical. In practice it is a niche option, available in some open-source stacks and rarely exposed by hosted APIs, but it is a clean illustration that "truncate the tail" is not the only philosophy available.

The pattern across all four: they run before the final random draw, and they all end by renormalizing the survivors back to a sum of 1. Temperature reshapes; these methods prune. You can — and inference stacks routinely do — stack several: apply top-k, then top-p, then temperature, then sample. Order matters at the margins, and different libraries chain them differently, which is one more reason a magic number copied from one tool rarely transfers cleanly to another.

Repetition, frequency, and presence penalties

Temperature and the truncation methods all treat each step in isolation — they look only at the current distribution and ignore what the model has already written. That is why, at low temperature especially, models fall into loops: "the best the best the best," or a paragraph that restates itself three times. A separate family of controls exists to fight that, and they work by editing the logits based on history before softmax runs.

• Repetition penalty divides (or on some implementations subtracts from) the logit of any token that has already appeared in the context. A value of 1.0 is off; 1.1–1.2 is a mild nudge; push it past ~1.3 and you start seeing the opposite pathology, where the model avoids necessary words — refusing to reuse "the" or a subject's name — and the prose turns stilted.
• Frequency penalty (an OpenAI-style additive control, typically −2.0 to 2.0) subtracts an amount from a token's logit proportional to how many times it has already occurred. The more often a word has appeared, the harder it is penalized. This scales with repetition, so it is good at suppressing a word that is genuinely being overused.
• Presence penalty (same −2.0 to 2.0 range) subtracts a flat amount from any token that has appeared even once, regardless of count. It does not care about frequency — it cares about novelty, gently pushing the model toward tokens it has not used yet, which nudges the output to cover new topics rather than dwell.

The distinction between the last two is the one people get wrong: frequency penalty punishes repetition of the same word, presence penalty punishes staying on the same ground at all. Use frequency penalty when a model keeps parroting a specific phrase; use presence penalty when you want it to range across more subject matter. Both are blunt — they operate on raw tokens, not meaning, so they cannot tell a legitimately necessary repeat ("Section 3.1... Section 3.2...") from a degenerate one. For most tasks, leaving them at zero and fixing loops with a better prompt or a lower-quality-tolerant top-p is cleaner than reaching for these knobs. They earn their keep mainly in long-form generation where loop-collapse is a real risk.

Beam search, and why chat models abandoned it

There is an entire decoding strategy the chat era quietly walked away from, and understanding why sharpens the intuition for everything above. Beam search does not sample at all. Instead of committing to one token and moving on, it keeps the b most promising partial sequences (the "beams") alive at once, extends each by every candidate next token, scores the resulting longer sequences by their cumulative probability, and prunes back to the best b. At the end it returns the highest-probability whole sequence it found — not just a greedy chain of locally-best tokens.

For tasks with a narrow, well-defined correct output, that is genuinely powerful. Beam search was the workhorse of machine translation and speech recognition for years, because there the goal is to find the single most probable rendering and greedy decoding can be led astray by a locally-attractive wrong turn. Optimizing over sequences instead of tokens finds better global answers.

So why don't ChatGPT-style models use it? Because for open-ended generation, the single most probable sequence is a bad target. Maximizing total probability systematically favors short, safe, generic, repetitive text — the model discovers that "I'm not sure I can help with that" and endless bland hedging score high, and beam search dutifully hunts those down. The output becomes flat and often degenerates into loops. Human-sounding text lives in the variety of the distribution, not at its single peak, which is the counterintuitive result that pushed the field toward sampling (top-p was introduced specifically to address the degeneration that greedy and beam search produce on open-ended text). Beam search is also expensive — you run the model across b sequences in parallel every step — and it does not fit the token-by-token streaming that chat interfaces depend on, since you cannot show a token until you are sure a later beam will not overtake it. For anything creative or conversational, deliberately sampling from a truncated distribution beats hunting for the mathematically-most-likely paragraph. The most likely paragraph is boring by construction.

A quick comparison

Setting	What it changes	Effect of raising it	Effect of lowering it	Best mental model
Temperature	Shape of the whole distribution (before softmax)	Flatter → more varied, more errors	Sharper → more repetitive, more predictable	Contrast dial
Top-p (nucleus)	Which candidates are eligible, by probability mass	Wider candidate pool, more variety	Narrower pool, safer picks	Adaptive shortlist
Top-k	Which candidates are eligible, by fixed count	More candidates allowed	Fewer candidates allowed	Fixed-size shortlist

Note the asymmetry: raising temperature and lowering top-p push in opposite directions on variety. If you crank temperature to 1.5 but also set top-p to 0.5, the aggressive temperature only ever applies within a tightly clipped candidate set. People who combine extreme values often get results that feel contradictory because the two knobs are fighting.

When to turn it down, when to turn it up

The honest answer is task-dependent, and it maps cleanly onto one question: do you want the safe answer or the surprising one?

Turn it down (low temperature, tight top-p) when there is a right answer and you want it reliably:

• Extracting structured data, filling a JSON schema, or returning a fixed format.
• Code generation and editing, where an off-distribution token is a syntax error. (More on the workflows in AI coding agents.)
• Classification, routing, yes/no decisions.
• Factual Q&A and summarization where you want faithfulness over flair.
• Anything downstream of a parser that will choke on variation.

Turn it up (higher temperature, generous top-p) when you want range and there is no single correct output:

• Brainstorming, naming, and idea generation where you'll cherry-pick.
• Fiction, dialogue, and stylistic writing that would feel robotic if identical every time.
• Generating diverse variations to compare — say, ten different subject lines.
• Any case where sameness is the failure mode.

A practical habit: default low, raise only when you see a specific problem. If outputs feel wooden, bump temperature. If outputs drift, hallucinate, or break format, drop it. Change one knob at a time; changing both makes cause and effect impossible to read. And remember these settings don't rescue a weak prompt — if the model keeps missing what you want, writing a clearer prompt will move the needle far more than any sampling tweak.

Settings per task: a concrete cheat sheet

Numbers are dangerous in an evergreen post — defaults drift, providers rename things, and a value that suits one model's native distribution misfits another's. Treat the following as starting points and reasoning, not gospel, and always verify against the specific model's documentation. The value of a cheat sheet is less the digits than the logic for why each row sits where it does.

Task	Temperature	Top-p	Reasoning
Structured extraction / JSON	~0	1.0 (irrelevant at T≈0)	One correct parse; any deviation breaks a downstream parser. Kill variance.
Code generation and editing	0–0.3	~0.95	An off-distribution token is a syntax error. A hair of variety helps escape a bad local phrasing, but keep it tight.
Classification / routing / yes-no	~0	1.0	You want the single most probable label, reproducibly.
Factual Q&A, summarization	0.2–0.5	~0.9	Faithfulness over flair, but enough slack for natural phrasing.
Standard chat / explanation	0.7	0.9–0.95	The common "native" balance most chat models are tuned around.
Brainstorming, naming, ideation	0.9–1.1	~0.95	You will cherry-pick, so reward range; the top-p guardrail clips outright nonsense.
Fiction, dialogue, marketing copy	0.8–1.2	0.9–1.0	Sameness is the failure mode. Variety is the product.
Diverse variations (N subject lines)	0.9–1.1	~0.95	You explicitly want the runs to differ from each other.

A few principles that outlive any specific number. Change one knob at a time so cause and effect stay legible — moving temperature and top-p together makes a regression impossible to bisect. Default low and raise on evidence, not on a hunch that "more creative" is better; woodenness is a symptom you can see, whereas a silent factual error from too-high temperature is not. And when temperature is near 0, top-p barely matters, because the distribution is already a spike — there is nothing in the tail left to clip. The two knobs are most worth tuning together precisely in the middle of the range, where the distribution has real spread. Above all, remember that none of these settings rescue a weak prompt; writing a clearer prompt moves the needle far more than any sampling tweak.

Sampling and reasoning models

Many of today's reasoning models — the ones that generate a long internal chain of thought before committing to an answer — behave oddly under manual temperature control, and several providers recommend leaving the setting at its default or restrict it outright. It is worth understanding why, because it is not arbitrary caution.

A reasoning model's quality depends on a long chain of intermediate tokens, and errors compound across that chain. If each step has a small chance of taking a bad token, a chain hundreds of tokens long accumulates that risk multiplicatively — a temperature that looks harmless for a one-sentence reply can meaningfully degrade a long derivation. These models are typically post-trained with a specific sampling regime in mind, so the "native" behavior at the recommended setting is the one that was actually optimized. Override it and you are sampling from a distribution the training never tuned for.

There is a second, subtler point. Some reasoning systems generate multiple candidate chains and select among them, or use the diversity of sampling deliberately inside their own search. In those cases the randomness is not a user-facing creativity knob at all — it is machinery internal to how the model reaches an answer, and exposing it to you would just let you break it. This is why some providers hide or fix temperature for these models entirely. The practical rule is simple: with reasoning models, treat the default as load-bearing and do not touch the knob unless the documentation explicitly invites you to. If you need more variety, ask for it in the prompt ("give me three distinct approaches") rather than reaching for the sampler.

Why temperature 0 isn't fully deterministic

This one surprises people, and it's a good test of whether you actually understand the stack.

Setting temperature to 0 (greedy decoding) means "always take the single highest-probability token." In pure math, that's deterministic: same input, same output, every time. In a real deployed system, it frequently isn't. Same prompt, same seed, same model — and you occasionally get a different completion. Why?

• Floating-point arithmetic isn't associative. (a + b) + c can differ from a + (b + c) in the last bits. When two candidate tokens have logits that are extremely close, a rounding difference can flip which one is "highest." The tie-break is decided by noise.
• Batching and parallelism change the order of operations. When your request is batched with others, or split across GPUs, the exact sequence of additions can vary run to run. Same math, different order, different last-bit result.
• Hardware and kernel choices vary. Different GPUs, driver versions, or optimized kernels can produce microscopically different numbers for the same operation. Usually invisible; occasionally decisive at a near-tie.

So "temperature 0" removes the deliberate randomness of sampling but not the incidental nondeterminism of the machinery underneath. Greedy decoding sharply reduces variance — it's the right choice when you want reproducibility — but treating it as a guarantee will eventually burn you. If you need bit-for-bit repeatability, you generally need a fixed seed plus deterministic-inference settings plus a pinned serving stack, and even then some providers won't promise it. This is also why cached results and fixed seeds matter for anyone tracking spend and consistency — a theme in AI inference cost economics.

The practical takeaway: greedy is for consistency, not certainty. Design systems that tolerate the rare flip rather than assuming it can't happen.

Seeds, reproducibility, and what you can actually promise

If greedy decoding is not a guarantee, what about the other lever people reach for — the seed? Sampling's randomness is not cosmic; it comes from a pseudo-random number generator, and a PRNG is fully determined by its starting seed. Fix the seed and, in principle, the same sequence of "random" draws happens every time, so the same prompt at the same temperature yields the same completion. Several APIs expose a seed parameter for exactly this reason, and it is the right tool when you want to reproduce a specific sampled (non-greedy) output — for a bug report, a regression test, or a demo you need to run twice identically.

But a seed only controls the sampling draw. It does nothing about the floating-point and ordering nondeterminism described above, which happens earlier, when the logits themselves are computed. So the honest hierarchy of reproducibility looks like this:

1. Same seed, same temperature — usually reproduces the output, most of the time, on the same serving stack. Good enough for most testing.
2. Add pinned model version and deterministic-inference settings — tightens it further. Providers that take reproducibility seriously often return a "system fingerprint" so you can detect when the backend changed underneath you and invalidated your assumption.
3. Bit-for-bit guarantee across arbitrary infrastructure — effectively unavailable from hosted APIs. The moment your request is batched with strangers on shared GPUs, the order of floating-point operations is out of your hands.

Two practical consequences. First, a seed is a convenience for reproducing a run, not a contract; if your test asserts exact string equality on a hosted model, expect it to flake eventually, and assert on structure or semantics instead. Second, if you genuinely need determinism — regulated environments, cached responses, evaluation harnesses that must be stable — the realistic path is to run open-weights models on hardware you control with deterministic kernels enabled, and even then to build in tolerance for the occasional near-tie flip. For anyone tracking spend and consistency together, this is the same reason caching and fixed seeds matter operationally — a theme in AI inference cost economics.

Common mistakes

• Treating temperature as a quality knob. It's a variance knob. Higher isn't "smarter" and lower isn't "dumber" — they trade predictability against range.
• Cranking temperature to fix boring output that's really a prompt problem. If the model won't stop giving you the same bland answer, the constraint is usually the instruction, not the sampler.
• Setting temperature high and top-p low (or vice versa) without realizing they interact. You get muddled results because the two knobs pull against each other.
• Assuming top-p and temperature exist on every endpoint identically. Naming, defaults, and even availability differ by provider and model, and reasoning models may ignore or reject the parameter. Read the specific docs; don't port a magic number across models.
• Expecting temperature 0 to be a hash function. It reduces randomness; it does not eliminate it in production.
• Chasing a "best" value. There's no globally optimal temperature. There's only the value that fits this task's tolerance for surprise.

FAQ

What does temperature do in AI, in one sentence? Temperature controls how much randomness the model uses when choosing each next word: low temperature makes it reliably pick the most likely option, while high temperature lets it take bigger chances on less-likely options, producing more varied but riskier text. It reshapes the probability distribution the model already computed — it doesn't add new knowledge.

What's the difference between temperature and top-p? Temperature reshapes the entire probability distribution — sharpening it (predictable) or flattening it (varied) — before a token is chosen. Top-p (nucleus sampling) instead truncates the distribution, keeping only the smallest set of top tokens whose probabilities sum to p and sampling from that set. Temperature adjusts boldness; top-p adjusts how many candidates are even eligible. They're often used together.

Should I use temperature or top-p — or both? Most production systems use both: a temperature to set overall variance and a top-p (commonly around 0.9–0.95) as a guardrail that clips the improbable tail. If you only touch one, temperature is the more intuitive lever. Avoid pushing both to extremes in opposite directions, since they interact and can produce contradictory-feeling output.

What temperature should I use for factual or coding tasks? Keep it low — near 0 for anything with a single correct answer, structured output, or code, where an off-distribution token becomes an error or a syntax break. Raise it only for open-ended work like brainstorming or creative writing, where you actually want variety and there's no single right output.

Why does temperature 0 still give different answers sometimes? Because real inference isn't pure math. Floating-point rounding, the order of operations under batching and parallelism, and differences across GPUs or kernels can flip which token counts as "highest" when two are nearly tied. Temperature 0 removes deliberate sampling randomness but not this incidental hardware-level nondeterminism, so greedy decoding gives consistency, not a guarantee.

Does higher temperature make the model smarter or more creative? No — it makes the model more variable, not more capable. It raises the odds of committing to a less-probable continuation the model was already considering. Sometimes that's a fresh phrasing; sometimes it's an error or broken grammar. The model's actual knowledge lives in the logits; sampling only chooses among options it already ranked, so no temperature setting can produce a good token that scored near zero.

What is min-p, and is it better than top-p? Min-p keeps every token whose probability is at least a fixed fraction of the top token's probability — for example, with min_p = 0.1 and a leading token at 0.6, only tokens above 0.06 survive. Because the cutoff scales with the model's confidence, min-p tends to hold up better than top-p at high temperatures, where top-p can accidentally admit junk once a long tail of weak tokens collectively crosses its mass threshold. "Better" depends on the task, but min-p is a reasonable default if you want to run higher temperatures without the output degrading into noise. Not every API exposes it.

What are frequency and presence penalties, and how do they differ? Both discourage repetition by editing logits before sampling, but along different axes. Frequency penalty subtracts more from a token the more times it has already appeared, so it targets a word that is being genuinely overused. Presence penalty subtracts a flat amount from any token that has appeared even once, nudging the model toward new vocabulary and new topics regardless of counts. Use frequency penalty to stop a parroted phrase; use presence penalty to push the model to range across more ground. Both are blunt token-level tools, so for most tasks leaving them at zero and fixing loops via the prompt is cleaner.

Why don't ChatGPT-style models use beam search? Beam search hunts for the single highest-probability whole sequence, which is excellent for narrow tasks like translation but wrong for open-ended chat: the most probable paragraph is systematically short, generic, and repetitive, and beam search reliably finds exactly that blandness, often collapsing into loops. Human-sounding text lives in the variety of the distribution, not at its peak, so chat models sample from a truncated distribution instead. Beam search also fights token-by-token streaming and costs more compute, which seals the case for conversational use.

Does setting a seed make the output fully reproducible? Not by itself. A seed fixes the pseudo-random draw used during sampling, so on the same serving stack the same prompt and temperature will usually reproduce. But it does nothing about the floating-point and operation-ordering nondeterminism that happens earlier, when logits are computed — and on shared, batched, hosted infrastructure that ordering is outside your control. A seed is a convenience for reproducing a run, not a hard guarantee; if you need bit-for-bit determinism, run open-weights models on hardware you control with deterministic kernels, and still tolerate the rare near-tie flip.

Should I change temperature on a reasoning model? Usually not. Reasoning models generate long internal chains where token errors compound across hundreds of steps, and they are post-trained around a specific sampling regime, so the default is the setting that was actually optimized. Some providers restrict or hide the knob for these models entirely because the randomness is machinery internal to how they reach an answer. If you want more variety, ask for it in the prompt ("give me three distinct approaches") rather than raising the temperature.