Concept Walkthrough

Two kinds of tokens

• Input tokens: everything in the request. System prompt, chat history, documents, tool definitions, the user's message.
• Output tokens: everything the model generates. The visible answer, tool call arguments, and, on reasoning models, hidden thinking tokens.

Providers bill these separately, and the usage object on each response reports them separately.

Why they behave differently

A useful analogy: input is like parsing a whole JSON file in one go, output is like a for loop that can only start iteration N after iteration N minus 1 finishes.

The model can process all input tokens in a batch, because they are all known up front. Output is generated autoregressively: one token, append it, run again for the next token. That sequential loop is slower and more expensive to serve per token.

Pricing

Output tokens cost more per token than input tokens. As of this writing, Anthropic's pricing page lists output at 5x the input price for every current Claude model. Other providers also price output above input, with their own ratios. Two more details matter:

• Cached input can be much cheaper. With prompt caching, repeated prefixes (a long system prompt, a fixed document) are billed at a fraction of the base input price on reads, with a surcharge on the first write.
• Reasoning tokens are output. OpenAI's docs say hidden reasoning tokens are billed as output, and Anthropic counts thinking tokens toward max_tokens. A short visible answer can still cost a lot.

Latency

Split latency into two numbers:

1. Time to first token (TTFT): how long before text appears. A longer input means more to process first, so TTFT grows with prompt size. Caching can help here too.
2. Generation time: how long the answer takes to finish. This grows with the number of output tokens, because they come one by one.

Anthropic's latency guide sums up the rule: fewer tokens to process and generate means a faster response. Streaming does not reduce total time, but it shows text as soon as the first token arrives, which is why chat UIs stream.

Key Code Explained

import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic();

// Keep prices in server config, copied from the provider's pricing page.
type Price = { inputPerMTok: number; outputPerMTok: number; cacheReadPerMTok: number; cacheWritePerMTok: number };

export function costUSD(usage: Anthropic.Usage, p: Price): number {
  const perToken = (perMTok: number) => perMTok / 1_000_000;
  return (
    usage.input_tokens * perToken(p.inputPerMTok) +
    (usage.cache_read_input_tokens ?? 0) * perToken(p.cacheReadPerMTok) +
    (usage.cache_creation_input_tokens ?? 0) * perToken(p.cacheWritePerMTok) +
    usage.output_tokens * perToken(p.outputPerMTok)
  );
}

export async function timedAnswer(model: string, prompt: string, price: Price) {
  const start = performance.now();
  let ttft: number | null = null;

  const stream = client.messages.stream({
    model,
    max_tokens: 800, // hard cap on output: bounds both cost and generation time
    messages: [{ role: 'user', content: prompt }],
  });

  stream.on('text', () => {
    if (ttft === null) ttft = performance.now() - start; // first visible text
  });

  const message = await stream.finalMessage();
  const total = performance.now() - start;

  return {
    ttftMs: ttft,
    totalMs: total,
    inputTokens: message.usage.input_tokens,
    outputTokens: message.usage.output_tokens,
    cost: costUSD(message.usage, price),
  };
}

Anthropic reports total input as the sum of input_tokens, cache_creation_input_tokens and cache_read_input_tokens, so the cost function prices each part at its own rate. Logging ttftMs and totalMs side by side tells you which half of the problem you have: a slow first token points at prompt size, model choice or missing caching; a slow finish points at answer length.

Tradeoffs

What Interviewers Actually Check

• Whether you know output is priced higher and why
• Whether you separate time to first token from total time
• Whether you know hidden reasoning tokens are billed as output
• Whether you read the usage object instead of estimating
• Whether you pick the right lever for the specific bottleneck

Follow-Up Questions

1. How would you show a cost estimate before the user clicks "Generate"?
2. Why does a long chat history slow down the first token on every turn?
3. How would you structure a prompt so prompt caching actually hits?
4. When is it worth calling a smaller model first and a larger model only on hard cases?
5. Which metric would you put on a dashboard for an AI feature: average or p95 TTFT? Why?

Common Candidate Mistakes

• Pricing a feature from prompt size alone
• Saying streaming makes the model generate faster
• Forgetting reasoning or thinking tokens when budgeting
• Shrinking the prompt to fix slow long answers, when output length is the real cause
• Computing cost in the browser with prices shipped in the client bundle

Interview Readiness Checklist

Before you leave this question, make sure you can answer:

• Can you explain why output tokens are generated one at a time?
• Can you say which token type usually dominates cost for long answers?
• Can you compute cost from a usage object, including cached tokens?
• Can you measure TTFT and total time in TypeScript?
• Can you pick an optimization for a slow first token versus a slow finish?

Summary

Input tokens are what you send and can be processed together; output tokens are generated one by one. That is why output is priced higher per token (Anthropic currently lists 5x input for its Claude models) and why answer length drives total response time. Input size mainly affects cost and time to first token, and prompt caching can make repeated input much cheaper. Hidden reasoning tokens count as output. In practice, cap and shorten outputs, trim or cache inputs, stream for perceived speed, and always compute cost from the usage object returned with each response.