How a Chat AI Handles Your Message, End to End: Prompt Assembly, KV Cache and Streaming

Architecture reconstructed from public sources. See Sources at the end.

You type "why is my useEffect running twice?" into a chat app and press Enter. Within a moment, words start appearing. In this case study we follow that one message the whole way: from the textarea, through the API server, into the model's GPU, and back to the screen as a stream.

Monday's post, LLMs for Frontend Engineers: Tokens, Context Windows and Cost Explained, showed that the model is stateless and every turn resends the conversation. This post shows what the system around the model does with that fact.

TL;DR

A chat AI is a thin, stateful product layer wrapped around a stateless model. On every message the server rebuilds the full prompt (instructions, tools, history, your new text), the inference engine processes that prompt in one parallel pass called prefill, then generates the answer one token at a time in a loop called decode, and each token is streamed to the browser as a server-sent event. The design decision that makes it all affordable is reusing work on the shared prefix: the KV cache avoids recomputing earlier tokens inside one response, and prompt caching avoids recomputing the same prefix across turns. That is why the order of your prompt matters as much as its content.

What the user sees

From the browser, the behaviour looks simple:

1. You press Enter. Your message appears in the thread immediately (an optimistic update, like adding a todo before the API confirms it).
2. A typing indicator shows for a short pause. This pause is the time to first token.
3. Text starts appearing in small bursts, a few characters at a time, with markdown formatting settling in as it arrives.
4. A Stop button is available while text is streaming. Pressing it freezes the answer where it is.
5. Long conversations still "remember" earlier messages, until one day the app says the conversation is too long, or quietly drops older context.

In DevTools you usually see one long-lived POST whose response keeps growing. That tells you the transport is a stream, not polling.

Requirements

Here is how an interviewer would frame it.

Functional

• Send a message and receive a streamed answer in the same conversation.
• Keep multi-turn context: the model must see earlier messages.
• Let the user stop generation at any time.
• Persist conversations so a reload shows the full thread.
• Report token usage so the product can meter and bill.

Non-functional

• Latency, split in two. Time to first token (TTFT) and time per output token (TPOT) are different problems with different fixes. The DistServe paper names exactly these two metrics as the latency targets for serving.
• Cost. Every turn resends the whole history, so cost grows with conversation length. The design must avoid paying full price for repeated tokens.
• Scale. Many users share a small number of very expensive accelerators. GPU utilisation is the main cost lever.
• Reliability. Streams can break halfway. Overload must surface as a clean error, and a stopped stream must stop spending compute.
• Consistency of quality. Requests may run on different hardware. Anthropic's 2025 postmortem notes Claude runs on AWS Trainium, NVIDIA GPUs and Google TPUs, and each change must be validated on all of them.

High-level architecture

Diagram
100%
flowchart LR
    UI["Browser chat UI<br/>(React)"] -->|"POST /chat<br/>chatId + text"| API["Chat API server"]
    API --> STORE[("Conversation store")]
    API --> CTX["Context builder<br/>instructions, tools,<br/>history, new message"]
    CTX --> GW["Model API / router"]
    GW --> SCHED["Inference scheduler<br/>continuous batching"]
    SCHED --> GPU["Accelerators<br/>prefill + decode"]
    GPU <--> KV[("KV cache<br/>+ prefix cache")]
    GPU -->|"tokens"| GW
    GW -->|"SSE events"| API
    API -->|"SSE events"| UI
    API -->|"save answer + usage"| STORE
Zoom or open fullscreen for a closer look
visualized by

Walk through it left to right:

• Browser chat UI. Owns input state, the optimistic message, the streaming render and the Stop button. It sends only the new message and a conversation id.
• Chat API server. This is the part you would build in Next.js or Node. It authenticates the user, loads the conversation, calls the model, relays the stream and saves the result. This is the "stateful" layer.
• Context builder. Turns the stored conversation into the exact prompt the model sees. Because the model has no memory, this step runs on every single turn.
• Model API and router. The provider's front door. It validates the request, applies rate limits and routes it to a pool of servers that can handle the model and the context length.
• Inference scheduler. Groups requests from many users into batches so one pass over the model weights serves many people at once.
• Accelerators and KV cache. Where prefill and decode actually run. The KV cache holds intermediate attention data so earlier tokens are not recomputed.

Deep dives

1. Context assembly and prompt caching

Think of the model as a pure function: answer = model(prompt). There is no this.history inside it. The chat server keeps state and rebuilds the argument each time, the same way a React component re-renders from props and state instead of mutating the DOM.

The rebuilt prompt has a stable order. Anthropic's prompt caching docs describe it as tools, then system, then messages, and caching covers everything up to a marked breakpoint. OpenAI's docs say a cache hit needs the entire rendered prefix to match. So the prompt looks like this:

Diagram
100%
flowchart LR
    A["Tool definitions<br/>(rarely change)"] --> B["System instructions<br/>(rarely change)"]
    B --> C["Turn 1 ... Turn N-1<br/>(grows, but old turns<br/>never change)"]
    C --> D["Turn N: new message<br/>(always new)"]
    style A fill:#1f3d2b,stroke:#3ecf8e
    style B fill:#1f3d2b,stroke:#3ecf8e
    style C fill:#1f3d2b,stroke:#3ecf8e
    style D fill:#3d1f1f,stroke:#cf3e3e
Zoom or open fullscreen for a closer look
visualized by

Everything in green was already processed on the previous turn. Prompt caching lets the provider store the processed state of that prefix and reuse it:

• Anthropic: you mark a breakpoint with cache_control, or use automatic caching, which moves the breakpoint forward as the conversation grows. The default lifetime is 5 minutes, refreshed for free each time it is used, with an optional 1 hour TTL. Cache reads are billed at a fraction of the base input price (0.1x by default, lower on some newer models), and cache writes cost more (1.25x for the 5 minute TTL).
• OpenAI: caching is on by default for supported models. The current docs give a 1,024 token minimum for GPT-5.6 and later, cache reads at 0.1x the uncached input rate, and a prompt_cache_key to help route related requests to the same cache.

Both report cache hits in the usage object (cache_read_input_tokens for Anthropic, input_tokens_details.cached_tokens for OpenAI). The frontend lesson is concrete: put stable content first and never mutate it. Injecting the current time or the user's name at the top of the system prompt changes the prefix on every turn and throws away the cache. It is the same idea as a stable key prop: change it and React throws away the subtree.

When history no longer fits the context window, the builder must trim or summarise older turns. Every trim changes the prefix, so trimming is a cost event too. Good builders trim in large steps rather than one message at a time.

2. Inside the inference engine: prefill, decode and batching

Once the prompt reaches the accelerator, work happens in two phases. NVIDIA's inference optimisation guide explains them well:

• Prefill. All input tokens are known up front, so the model processes them in parallel as large matrix-matrix operations. This phase keeps the hardware busy. Its duration is most of your time to first token, and it grows with prompt length.
• Decode. The model generates one token at a time, and each token depends on the previous ones. This is closer to matrix-vector work. NVIDIA's guide calls it "a memory-bound operation", because each step mostly waits on reading weights and cached data from memory.

The KV cache is what makes decode tolerable. During attention, each token produces key and value vectors. The engine stores them in GPU memory so step 500 does not recompute steps 1 to 499. A frontend analogy: it is useMemo for every token you have already seen, keyed by position.

The KV cache is large and its size depends on sequence length, which you do not know in advance. The vLLM team found existing systems wasted 60% to 80% of KV cache memory through fragmentation and over-reservation. Their fix, PagedAttention, splits the cache into fixed-size blocks that do not need to be contiguous, mapped through a block table, just like virtual memory pages in an operating system. Waste drops to the last partly filled block, under 4% in their measurements. vLLM's automatic prefix caching builds on these blocks: each full block is hashed together with its parent block's hash, so two requests with the same prefix share the same physical blocks. That is prompt caching seen from the inside.

Batching is how the cost is shared. The naive approach runs a batch of requests until all of them finish, so a short answer waits for a long one. The Orca paper (OSDI 2022) introduced iteration-level scheduling: the scheduler decides batch membership at every decode step, so finished requests leave immediately and new ones join. NVIDIA calls the same idea in-flight batching. Your request joins a running batch instead of waiting for a new one.

Prefill and decode compete for the same hardware. A big prefill from one user can slow decode steps for everyone in the batch. DistServe proposes running the two phases on different GPUs so each can be tuned for TTFT or TPOT separately. Whether a given commercial provider does this is not public, so treat it as a known option, not a confirmed detail.

Routing sits in front of all of this. Anthropic's postmortem gives a rare public view: some short-context requests were misrouted to servers configured for a 1M token context window, and because routing was sticky, follow-up messages often went to the same wrong server. Two design facts fall out of that incident: servers are pooled by configuration such as context length, and routing is deliberately sticky, which also helps cache reuse.

3. The frontend: streaming transport and rendering

The provider streams tokens as server-sent events. Anthropic's streaming docs define the sequence: message_start, then for each content block a content_block_start, many content_block_delta events, a content_block_stop, then message_delta (carrying stop_reason and cumulative usage) and finally message_stop. Streams can also carry ping events and error events such as overloaded_error, and the docs say to handle unknown event types gracefully.

Your Node server usually does not forward these raw. It parses them (often through an SDK), keeps the full text to save at the end, and re-emits a simpler protocol to the browser. On the client, three problems matter:

1. Transport. EventSource only supports GET, and chat needs a POST body. So chat UIs read the stream with fetch, pipe response.body through a TextDecoderStream, and parse event: and data: lines themselves. Chunks do not align with events, so the parser must buffer until it sees a blank line.
2. Rendering. Tokens can arrive faster than the screen refreshes. Calling setState per token re-renders the message list for every chunk. Accumulate text in a ref and flush to state once per animation frame, so React renders at most once per frame. If the markdown renderer re-parses the whole message on each flush, memoise finished paragraphs and re-parse only the last one.
3. Cancellation. The Stop button calls abort() on an AbortController. The fetch closes, the server sees the connection close, and it aborts its upstream model call so decode stops spending compute. Without that last step, the user sees a stopped answer while you still pay for the full one.

Data flow: one message, end to end

Diagram
100%
sequenceDiagram
    participant U as Browser (React)
    participant S as Chat API (Node)
    participant DB as Conversation store
    participant M as Model API
    participant E as Inference engine
    U->>U: optimistic render of user message
    U->>S: POST /chat {chatId, text}
    S->>DB: load history for chatId
    S->>S: build prompt (tools, instructions, history, new text)
    S->>M: create message, stream: true
    M->>E: route to server pool, join batch
    E->>E: prefill (prefix cache hit for old turns)
    E-->>M: first token
    M-->>S: message_start, content_block_delta
    S-->>U: event: delta
    loop decode, one token per step
        E-->>M: next token
        M-->>S: content_block_delta
        S-->>U: event: delta (rendered once per frame)
    end
    M-->>S: message_delta (stop_reason, usage), message_stop
    S->>DB: save assistant message + usage
    S-->>U: event: done {usage}
Zoom or open fullscreen for a closer look
visualized by

1. The user presses Enter. React appends the user message to local state right away and shows a typing indicator.
2. The client sends a POST with the conversation id and the new text only. It keeps an AbortController for the Stop button.
3. The API server authenticates the request and loads the stored history for that conversation.
4. The context builder assembles the prompt in a stable order and trims history if it would exceed the context budget.
5. The server calls the model API with streaming enabled.
6. The provider routes the request to a server pool that can handle the model and context length, and the scheduler adds it to a running batch.
7. Prefill runs over the prompt. If the prefix matches a cached one, that part is reused instead of recomputed.
8. Decode produces the first token. The provider emits message_start and the first content_block_delta.
9. The API server relays each delta to the browser as its own SSE event and appends it to a server-side buffer.
10. The client parser buffers chunks, splits complete events, and writes text into a ref. A requestAnimationFrame callback flushes it into state.
11. Decode continues until a stop condition. The provider sends message_delta with stop_reason and usage, then message_stop.
12. The server saves the full assistant message and usage, then sends a final done event. The client clears the typing state.

Trade-offs

Confirmed vs inferred

Confirmed by sources

• Prompt caching covers tools, system and messages in that order, with a 5 minute default TTL and an optional 1 hour TTL (Anthropic docs).
• OpenAI caching is on by default, needs the full prefix to match, and reports cached_tokens in usage (OpenAI docs).
• The streaming event sequence, ping and error events, and cumulative usage in message_delta (Anthropic streaming docs).
• Prefill is parallel and compute-heavy; decode is sequential and memory-bound; the KV cache stores keys and values of earlier tokens (NVIDIA).
• PagedAttention, block-based KV cache and the 60% to 80% waste finding (vLLM blog); prefix blocks hashed with their parent hash (vLLM docs).
• Iteration-level scheduling (Orca paper) and prefill/decode disaggregation as a research design (DistServe paper).
• Claude runs on Trainium, NVIDIA GPUs and TPUs, routing is pooled by context configuration, and was sticky in 2025 (Anthropic postmortem).

Inferred or reconstructed

• That consumer chat apps such as ChatGPT or Claude.ai use the same pipeline as their public APIs internally. Likely, but not documented.
• The exact component split (chat API, context builder, router, scheduler) in any specific product.
• Whether a given commercial provider disaggregates prefill and decode, or uses vLLM-style paging.
• How a specific app trims or summarises long conversations.
• That chat frontends batch renders per animation frame. This is a common React pattern, not a documented detail of a named product.

Build a mini version in JS/TS

Three files: a Node server that rebuilds the prompt and streams SSE, a client that parses the stream, and a React hook that renders it per frame. This uses AI SDK 7 (Node 22+, ESM only), where system is now called instructions.

// server.ts   npm i ai @ai-sdk/openai   run: npx tsx server.ts
import { createServer, type IncomingMessage } from 'node:http';
import { streamText, type ModelMessage } from 'ai';
import { openai } from '@ai-sdk/openai';

const INSTRUCTIONS = 'You are a concise assistant for frontend engineers.'; // stable prefix
const chats = new Map<string, ModelMessage[]>(); // demo store; use a database in production

const readJson = async (req: IncomingMessage) => {
  let body = '';
  for await (const chunk of req) body += chunk;
  return JSON.parse(body);
};

createServer(async (req, res) => {
  const { chatId, text } = await readJson(req);
  const history = [
    ...(chats.get(chatId) ?? []),
    { role: 'user', content: text } as ModelMessage,
  ];

  const controller = new AbortController();
  res.on('close', () => {
    if (!res.writableEnded) controller.abort();
  }); // Stop button
  res.writeHead(200, {
    'Content-Type': 'text/event-stream',
    'Cache-Control': 'no-cache',
    'Access-Control-Allow-Origin': '*',
  });
  const send = (event: string, data: unknown) =>
    res.write(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`);

  const result = streamText({
    model: openai('gpt-5-mini'),
    instructions: INSTRUCTIONS,
    messages: history,
    maxOutputTokens: 800,
    abortSignal: controller.signal,
  });

  try {
    for await (const delta of result.textStream) send('delta', { text: delta });
    chats.set(chatId, [
      ...history,
      { role: 'assistant', content: await result.text },
    ]);
    send('done', await result.usage); // includes inputTokenDetails.cacheReadTokens
  } catch {
    if (!controller.signal.aborted)
      send('error', { message: 'generation failed' });
  }
  res.end();
}).listen(3000);

// client.ts   works in the browser and in Node 22+
type Usage = Record<string, unknown>;

export async function sendMessage(
  chatId: string,
  text: string,
  onDelta: (t: string) => void,
  signal?: AbortSignal,
): Promise<Usage | undefined> {
  const res = await fetch('http://localhost:3000/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ chatId, text }),
    signal,
  });
  if (!res.ok || !res.body) throw new Error(`HTTP ${res.status}`);

  const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
  let buffer = '';
  for (;;) {
    const { value, done } = await reader.read();
    if (done) return;
    buffer += value;
    const frames = buffer.split('\n\n');
    buffer = frames.pop() ?? ''; // keep the incomplete frame for the next chunk
    for (const frame of frames) {
      const event = frame.match(/^event: (.*)$/m)?.[1];
      const data = frame.match(/^data: (.*)$/m)?.[1];
      if (!data) continue;
      const payload = JSON.parse(data);
      if (event === 'delta') onDelta(payload.text);
      if (event === 'error') throw new Error(payload.message);
      if (event === 'done') return payload;
    }
  }
}

// useStreamingReply.ts   one React render per animation frame, not per token
import { useRef, useState, useCallback } from 'react';
import { sendMessage } from './client';

export function useStreamingReply(chatId: string) {
  const [reply, setReply] = useState('');
  const [streaming, setStreaming] = useState(false);
  const buffer = useRef('');
  const frame = useRef(0);
  const controller = useRef<AbortController | null>(null);

  const flush = () => {
    frame.current = 0;
    setReply(buffer.current);
  };

  const send = useCallback(
    async (text: string) => {
      controller.current = new AbortController();
      buffer.current = '';
      setReply('');
      setStreaming(true);
      try {
        await sendMessage(
          chatId,
          text,
          (t) => {
            buffer.current += t;
            if (!frame.current) frame.current = requestAnimationFrame(flush);
          },
          controller.current.signal,
        );
      } finally {
        cancelAnimationFrame(frame.current);
        flush();
        setStreaming(false);
      }
    },
    [chatId],
  );

  const stop = () => controller.current?.abort();
  return { reply, streaming, send, stop };
}

Run the server, call sendMessage('demo', 'Hi') twice from a script, and log the done payload. Once the stable prefix is long enough to pass the provider's minimum, you can watch cached input tokens appear in the usage on later turns.

Interview angle

If the prompt is "Design the frontend of a ChatGPT-style chat app", walk through it like this:

1. Clarify scope. Text only or files too? Multi-device sync? Do we own the model or call a provider API? State the two latency metrics up front: time to first token and time between tokens.
2. Draw the request path. Browser, chat API, conversation store, model API. Say clearly that the model is stateless, so the server rebuilds the prompt every turn.
3. Pick the transport and justify it. SSE over a POST fetch for one-way token streaming; mention why EventSource does not fit (GET only) and when WebSockets would (real-time collaboration, server-initiated pushes).
4. Design client state. A message list keyed by id, an optimistic user message, a streaming assistant message held in a ref, flushed once per frame. Explain how you avoid re-rendering the whole list.
5. Handle failure and cancellation. Abort on Stop and on unmount, propagate the abort to the model call, show a retry for error events, and keep partial text visible.
6. Talk cost. Stable prompt prefix for caching, history trimming or summarising near the context limit, maxOutputTokens caps, and usage logged per message.
7. Close with trade-offs. Name one change you would make at scale, such as resumable streams for flaky mobile networks.

Sources

• Claude Docs: Streaming Messages (accessed October 2026)
• Claude Docs: Prompt caching (accessed October 2026)
• OpenAI: Prompt caching guide (accessed October 2026)
• Anthropic Engineering: A postmortem of three recent issues (September 17, 2025)
• NVIDIA Technical Blog: Mastering LLM Techniques, Inference Optimization (November 2023, updated December 2025)
• vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (June 20, 2023)
• vLLM Docs: Automatic Prefix Caching design (accessed October 2026)
• Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022
• DistServe: Disaggregating Prefill and Decoding, OSDI 2024
• Vercel Changelog: AI SDK 7 is now available (June 25, 2026)
• AI SDK Reference: streamText (accessed October 2026)
• AI SDK: Migrate 6.x to 7.0 (accessed October 2026)
• MDN: TextDecoderStream
• MDN: Using server-sent events