Concept Walkthrough

The window is shared by everything

The context window is the token budget for a single request. Everything the model sees and everything it writes back must fit inside it:

• the system prompt
• every earlier user and assistant message you resend
• tool definitions and tool results
• attached documents and images
• the tokens the model generates in its reply

Think of it like a fixed-height div with overflow: hidden. You can put anything inside, but once it is full, something has to go.

Why chats grow until they break

LLM APIs are stateless, like a REST endpoint with no session. Anthropic's context window docs describe each turn's input as all previous conversation history plus the current user message. So turn 1 sends one message, turn 20 sends twenty. Cost and latency grow every turn, and eventually the request does not fit.

What happens at the limit

This differs by provider, so check the docs for the one you use. On Anthropic's API:

• If the input alone exceeds the window, the API returns a 400 invalid_request_error.
• On Claude 4.5 and later, if the input fits but the model runs out of room while writing, the response comes back with stop_reason: "model_context_window_exceeded". The answer is cut short, not rejected.
• Separately, stop_reason: "max_tokens" means it hit the max_tokens you set.

Bigger is not automatically better

Even before you hit the hard limit, long contexts cost more on every request and take longer to process. Anthropic's docs also note that accuracy and recall degrade as the token count grows. So the real goal is to keep context relevant, not just under the limit.

The usual strategies

1. Sliding window: keep the system prompt plus the newest messages that fit a token budget.
2. Summarize older turns: replace early messages with a short summary message.
3. Retrieve instead of resend: store documents elsewhere and pull in only relevant parts (this is RAG, covered later in the series).
4. Provider features: Anthropic offers server-side compaction (in beta for newer models) that summarizes earlier parts of a conversation for you.

Key Code Explained

import Anthropic from '@anthropic-ai/sdk';

type Msg = Anthropic.MessageParam;
const client = new Anthropic();

// Keep the newest messages that fit in the budget. Never drop the latest user turn.
export async function trimHistory(
  model: string,
  system: string,
  messages: Msg[],
  inputBudget: number,
): Promise<Msg[]> {
  let kept = [...messages];

  while (kept.length > 1) {
    const { input_tokens } = await client.messages.countTokens({ model, system, messages: kept });
    if (input_tokens <= inputBudget) break;
    // Drop the oldest user + assistant pair so roles keep alternating
    kept = kept.slice(2);
  }
  return kept;
}

export async function reply(model: string, system: string, history: Msg[]) {
  const messages = await trimHistory(model, system, history, 50_000);
  const res = await client.messages.create({ model, system, max_tokens: 1024, messages });

  if (res.stop_reason === 'max_tokens' || res.stop_reason === 'model_context_window_exceeded') {
    // The answer was cut off. Tell the UI so it can show "Continue" instead of pretending it is complete.
    return { res, truncated: true, droppedTurns: history.length - messages.length };
  }
  return { res, truncated: false, droppedTurns: history.length - messages.length };
}

The budget is set well below the real window so the reply has room. Counting with the provider API in a loop is simple but adds calls; in production you would count once and estimate per message, or summarize in one step. droppedTurns lets the UI show a small "earlier messages were trimmed" note so users are not confused when the model forgets something.

Tradeoffs

What Interviewers Actually Check

• Whether you know the API is stateless and history is resent each turn
• Whether you count output tokens as part of the same window
• Whether you trim by tokens, not by message count
• Whether you handle both the rejected request and the cut-off answer
• Whether you think about the user experience when context is dropped

Follow-Up Questions

1. How would you decide the token budget for history versus the reply?
2. Where would you store the summary of older turns, and how do you keep it updated?
3. How would you handle a single user message that is larger than the window on its own?
4. What changes if the conversation includes tool calls and tool results?
5. How would you test that trimming never removes the system prompt?

Common Candidate Mistakes

• Saying "the model remembers the conversation" without mentioning the app resends history
• Keeping the last N messages regardless of their size
• Removing one message at a time and breaking user/assistant alternation
• Showing a cut-off answer as if it were complete
• Treating a bigger context window as a reason to stop managing context

Interview Readiness Checklist

Before you leave this question, make sure you can answer:

• Can you list everything that uses up the context window?
• Can you explain why the tenth turn costs more than the first?
• Can you name the error and stop reasons to handle on your provider?
• Can you write a token-based sliding window?
• Can you explain when you would summarize instead of trimming?

Summary

The context window is the total token budget for one request, shared by the system prompt, resent history, tools, documents and the generated answer. Because chat APIs are stateless, the app resends history every turn, so long conversations get slower, cost more and eventually exceed the window. On Anthropic's API an oversized input returns a 400 error, and on newer models a reply that runs out of room stops with a model_context_window_exceeded stop reason. Manage it deliberately: trim by tokens, summarize or compact older turns, retrieve large documents instead of resending them, and tell users when early context has been dropped.