Concept Walkthrough

Where the variation comes from

1. Sampling. The model picks each next token by sampling from a probability distribution. One different early token sends the rest of the answer down a different path, like a branch in a recursive function.
2. Temperature is not a lock. Lower temperature reduces variation, but Anthropic's API reference states that even temperature 0.0 is not fully deterministic. Some newer models also only accept the default temperature.
3. Hidden context changes. The model sees more than the user's text: system prompt, chat history, retrieved documents, tool results, today's date. If any of those differ, the answer can differ.
4. Model changes. Providers release new versions. If you call a moving alias, the model behind it can change. Pin the exact model ID where your provider offers one, and record it.

So for a UI engineer, the right mental model is: an LLM call is not a pure function. Same input does not guarantee same output.

What the UI should do

Treat generated text like user-generated content: once created, it is a record.

• Generate once, store, then render the stored copy. For a job post summary, generate it on publish or on first request, save it, and serve it from the database. Refreshing should never trigger a new generation.
• Store what the user saw. Chat messages, shared links and exports must point to stored output, not re-run prompts.
• Make variation a feature where it helps. In chat, offer "Regenerate" and keep earlier versions so users can switch back.
• Validate what the app depends on. If the UI needs a category, a score or JSON, validate it in code; never depend on exact wording.
• Record metadata. Model ID, prompt version and timestamp make it possible to explain later why an answer looks the way it does.

Key Code Explained

import { useState } from 'react';

type Version = {
  id: string;
  text: string;
  model: string; // exact model ID used
  promptVersion: string; // e.g. 'summary-v3'
  createdAt: string;
};

type AssistantMessage = { id: string; versions: Version[]; activeIndex: number };

export function AssistantBubble({
  message,
  onRegenerate,
}: {
  message: AssistantMessage;
  onRegenerate: (messageId: string) => Promise<void>; // server appends a new Version
}) {
  const [index, setIndex] = useState(message.activeIndex);
  const [busy, setBusy] = useState(false);
  const total = message.versions.length;
  const current = message.versions[index];

  async function regenerate() {
    setBusy(true);
    try {
      await onRegenerate(message.id);
      setIndex(total); // jump to the new version once it is added
    } finally {
      setBusy(false);
    }
  }

  return (
    <div>
      <p>{current.text}</p>
      <div aria-label='Response versions'>
        <button disabled={index === 0} onClick={() => setIndex(index - 1)}>Previous</button>
        <span>Response {index + 1} of {total}</span>
        <button disabled={index === total - 1} onClick={() => setIndex(index + 1)}>Next</button>
        <button disabled={busy} onClick={regenerate}>Regenerate</button>
      </div>
    </div>
  );
}

Each assistant message holds an array of versions instead of a single string. Regenerate appends rather than overwrites, so nothing the user saw disappears. The model and promptVersion fields are what you need when someone asks "why did the bot say this last week?". In a real app you would also persist activeIndex so shared links show the version that was selected.

Tradeoffs

What Interviewers Actually Check

• Whether you can name causes beyond "randomness" (context, model versions)
• Whether you know temperature 0 is not deterministic
• Whether you persist output instead of regenerating on every view
• Whether regenerate keeps history instead of overwriting
• Whether you separate UI tests (mocked model) from quality checks (evals)

Follow-Up Questions

1. When would you invalidate a stored summary and generate a new one?
2. How would you show users that a response came from an older model or prompt version?
3. How would you let users give feedback on one specific version?
4. How do shared chat links work if the conversation continues after sharing?
5. Which parts of an AI feature should never depend on model output at all?

Common Candidate Mistakes

• Blaming only temperature and ignoring prompt, context and model changes
• Generating in a useEffect on page load, so every refresh costs money and changes content
• Replacing the old answer on regenerate
• Parsing free text with string matching and breaking when wording shifts
• Writing exact-match tests against live model output

Interview Readiness Checklist

Before you leave this question, make sure you can answer:

• Can you name four sources of variation in LLM output?
• Can you explain why an LLM call is not a pure function?
• Can you design a message shape with versions and metadata?
• Can you explain when to cache versus when to regenerate?
• Can you describe how to test UI code that calls a model?

Summary

The same prompt can return different answers because models sample tokens, temperature 0 is still not fully deterministic, hidden context like history or retrieved data changes, and models get updated. Treat an LLM call as a non-pure function and treat its output as a record. Generate once and store for content that should stay stable, store exactly what users saw for chats and shares, offer regenerate with saved versions where variety helps, and validate any output the app logic depends on. Record the model ID and prompt version with each output so you can explain results later.