Llm fundamentals Ai Questions

At each step the model produces a probability for every possible next token, and temperature reshapes that distribution before one token is sampled. Lower temperature makes the likeliest tokens even more likely, so output is more focused and repeatable; higher temperature flattens it, so output is more varied. Lower it for extraction, classification and code, but check your model first: several newer models only accept the default value.

A token is the unit of text a model reads and writes: usually a word, part of a word, a punctuation mark or a space-plus-word chunk. Providers bill and rate-limit by tokens, not characters or words, so the same text can cost different amounts on models with different tokenizers. Count tokens with the provider's tokenizer or counting API instead of guessing from string length.

The context window is the maximum number of tokens a model can work with in one request: system prompt, chat history, documents, tools and the answer it generates. The model has no memory between calls, so your app resends the history every turn and it keeps growing. When the input is too large the API rejects the request, so the app must trim, summarize or compact older turns before that happens.

Input tokens are everything you send; output tokens are what the model generates. Output is priced higher per token (Anthropic currently lists it at 5x input for its Claude models), and it is generated one token at a time, so long answers dominate total response time. Long inputs cost more and delay the first token, but caching repeated prefixes can make that input much cheaper.

LLMs sample each next token from a probability distribution, so two runs of the same prompt can take different paths, and even temperature 0 is not fully deterministic. Model updates and changes to the hidden context (system prompt, history, retrieved data) change answers too. The UI should treat every response as a unique record: store what the user saw, offer regenerate instead of hiding variation, and validate anything the app depends on.