Concept Walkthrough

Generation is repeated sampling

An LLM writes one token at a time. At each step it scores every token in its vocabulary and turns those scores into probabilities. Then one token is sampled from that distribution, appended to the text, and the loop repeats.

If the model always picked the single most likely token, answers would be repetitive. Sampling adds controlled randomness.

What temperature does

Temperature reshapes the probability distribution before sampling:

• Lower temperature sharpens it. The top choices get more of the probability, so the model picks the "safe" token more often. Output is more focused and more repeatable.
• Higher temperature flattens it. Less likely tokens get a real chance, so output is more varied and creative, and also more likely to wander.

A React analogy: think of a weighted random picker for a "suggested item" carousel. Temperature controls how strongly you favour the highest-weighted item versus giving the long tail a turn.

When to lower it

Lower temperature fits tasks with one correct-ish answer:

• extracting fields from text
• classifying or tagging into fixed categories
• generating code or SQL
• short factual answers in a support bot

Keep the default or go higher for brainstorming, naming, marketing copy or anything where variety is the point. Anthropic's API reference gives the same guidance: closer to 0.0 for analytical or multiple choice tasks, closer to 1.0 for creative ones.

Two important caveats

1. Not deterministic. Anthropic states that even temperature 0.0 does not give fully deterministic results. Do not build features or tests that assume identical output.
2. Not always available. Anthropic's API reference marks temperature as deprecated: models released after Claude Opus 4.6 accept only the default of 1.0, and its latency guide suggests lowering effort instead. Some newer OpenAI reasoning models also reject the parameter. Treat temperature as a per-model capability, not a global setting.

When you cannot change temperature, get reliability from clearer prompts, examples, structured output with schema validation, and fixed choices in your own code.

Key Code Explained

import { generateText } from 'ai';

type ModelConfig = {
  id: string; // e.g. a 'provider/model' string for your gateway
  supportsTemperature: boolean; // keep this in your own config, per model
};

export async function tagJobPost(model: ModelConfig, description: string) {
  const { text } = await generateText({
    model: model.id,
    // Only send temperature to models that accept it, otherwise the call can fail
    ...(model.supportsTemperature ? { temperature: 0.2 } : {}),
    maxOutputTokens: 200,
    system: 'Return a comma separated list of skills from this list only: React, Next.js, TypeScript, Node.js, GraphQL.',
    prompt: description,
  });

  // Never trust free text blindly: filter against the allowed list in code
  const allowed = new Set(['React', 'Next.js', 'TypeScript', 'Node.js', 'GraphQL']);
  return text
    .split(',')
    .map((s) => s.trim())
    .filter((s) => allowed.has(s));
}

The AI SDK forwards temperature to the provider as is, and its docs say the valid range depends on the provider and model. That is why the flag lives in your model config. The final filter is the real safety net: even at low temperature the model can return something outside the list, so the code enforces the contract.

Tradeoffs

What Interviewers Actually Check

• Whether you explain temperature through next-token sampling, not as "creativity mode"
• Whether you can map settings to real product features
• Whether you know temperature 0 is not a determinism guarantee
• Whether you know some models reject the parameter
• Whether you add validation in code instead of relying on a parameter

Follow-Up Questions

1. What is top_p (nucleus sampling), and why do providers advise against tuning it together with temperature?
2. How would you get repeatable output for tests if temperature is not enough?
3. What is a seed parameter, and why is it only best effort on models that support it?
4. How would you A/B test two temperature settings for a chat feature?
5. Which would you change first for a hallucinating support bot: temperature or the prompt?

Common Candidate Mistakes

• Saying temperature controls how "smart" or "accurate" the model is
• Promising identical outputs at temperature 0
• Setting temperature on every request without checking the model accepts it
• Using high temperature for structured data extraction
• Not validating outputs because "temperature is low"

Interview Readiness Checklist

Before you leave this question, make sure you can answer:

• Can you explain next-token sampling in plain English?
• Can you describe the effect of low versus high temperature?
• Can you pick a setting for three different features and justify it?
• Can you explain why temperature 0 still varies?
• Can you explain how you would handle models that only accept the default?

Summary

The model generates text by sampling one token at a time from a probability distribution, and temperature reshapes that distribution. Lower values favour the likeliest tokens and give focused, more repeatable output; higher values flatten the distribution and give more varied output. Lower it for extraction, classification and code, keep the default or raise it for creative work. Two caveats matter in real projects: temperature 0 is still not fully deterministic, and several newer models (including Claude models released after Opus 4.6 and some OpenAI reasoning models) only accept the default. Treat temperature as a per-model option and get reliability from prompts, schemas and validation in code.