Streaming LLM Responses in Node.js: Disconnects, Backpressure and Mid-Stream Errors

Your streaming chat endpoint works in the demo. Then a user closes the tab halfway through a long answer, and your server keeps pulling tokens from the model until the answer is complete, and you pay for every one of them. Another request hangs forever because the provider stalled, and a third fails mid-answer, but the browser already received a 200 OK, so the UI just stops. This post is about streaming LLM responses in Node.js properly: the server half of the pipe, where those three bugs live.

If you have not read it yet, Building a Streaming AI Chat UI in React covers the browser side: SSE framing, reading response.body and rendering tokens. Here we stay on the server.

The concept: a stream is a subscription with two ends

You already know this pattern from React. A useEffect that subscribes to a WebSocket must return a cleanup function, or the subscription outlives the component. A streaming endpoint is the same thing with two subscriptions chained together:

• Upstream: your Node server subscribes to the model provider's token stream.
• Downstream: the browser subscribes to your server's SSE stream.

Every bug in this post is a missing cleanup or a missing signal between those two ends. The rules are:

1. When downstream goes away (tab closed, "Stop" clicked), cancel upstream. Otherwise you keep paying for tokens nobody reads.
2. When downstream is slow, stop reading upstream until it catches up. That is backpressure, and it keeps memory flat.
3. When upstream fails or stalls, tell downstream in-band, because the HTTP status is already sent.

Here is rule 1 as a sequence:

Diagram
100%
sequenceDiagram
  participant B as Browser
  participant S as Node server
  participant M as Model provider
  B->>S: GET /?q=... (SSE)
  S->>M: streamText (abortSignal)
  M-->>S: token deltas
  S-->>B: event: delta
  B--xS: tab closed / controller.abort()
  S->>S: res 'close' fires, response not ended
  S->>M: abort signal cancels request
  Note over S,M: onAbort runs, billing stops
Zoom or open fullscreen for a closer look
visualized by

Build it: an SSE endpoint that cleans up after itself

We will use the Vercel AI SDK (ai v7) with the OpenAI provider, plain node:http first, then a Next.js Route Handler. Install with npm i ai @ai-sdk/openai and npm i -D tsx, set OPENAI_API_KEY, and use a current Node LTS.

Step 1: one place for the model call

// llm.ts
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';

// One place for model, timeouts and cancellation wiring.
export function streamAnswer(prompt: string, abortSignal: AbortSignal) {
  return streamText({
    model: openai('gpt-5-mini'),
    prompt,
    abortSignal,
    timeout: { firstChunkMs: 15_000, chunkMs: 10_000 },
    onAbort: () => console.log('provider call cancelled'),
  });
}

Two settings matter here. abortSignal is how you cancel the provider request. timeout is newer and very useful for streams: per the AI SDK settings docs, firstChunkMs limits the wait for the first real output, and chunkMs limits the gap between output chunks after that. A timeout creates an abort signal internally, and if you pass both, whichever fires first wins. A stalled stream now ends on its own instead of hanging.

Step 2: map stream parts to SSE frames

result.stream yields typed parts: text-delta, finish, error, abort, plus tool and reasoning parts we ignore for now. We convert each one into an SSE frame using a named event: so the client can tell deltas from errors.

// sse.ts
import type { TextStreamPart, ToolSet } from 'ai';

export const SSE_HEADERS = {
  'Content-Type': 'text/event-stream; charset=utf-8',
  'Cache-Control': 'no-cache, no-transform',
  Connection: 'keep-alive',
  'X-Accel-Buffering': 'no', // tell nginx not to buffer this response
} as const;

const format = (event: string, data: unknown) =>
  `event: ${event}\ndata: ${JSON.stringify(data)}\n\n`;

// Map one AI SDK stream part to one SSE frame, or null to skip it.
export function toSseEvent(part: TextStreamPart<ToolSet>): string | null {
  switch (part.type) {
    case 'text-delta':
      return format('delta', { text: part.text });
    case 'finish':
      return format('done', {
        finishReason: part.finishReason,
        outputTokens: part.totalUsage.outputTokens,
      });
    case 'error':
      console.error('model error', part.error); // log the real error
      return format('error', { message: 'Generation failed. Please retry.' });
    case 'abort':
      return format('error', { message: 'Generation stopped.' });
    default:
      return null; // tool calls, reasoning, step events: not needed here
  }
}

This is the answer to "how do I send an error after a 200?". Once the first byte is written, the status code is final. Errors become data: an event: error frame the client renders as a retry message. The real error goes to your logs, never to the user.

Step 3: the Node server

// server.ts
import { createServer, type ServerResponse } from 'node:http';
import { once } from 'node:events';
import { streamAnswer } from './llm';
import { SSE_HEADERS, toSseEvent } from './sse';

// Backpressure: if the socket buffer is full, wait for 'drain'.
async function send(res: ServerResponse, frame: string, signal: AbortSignal) {
  if (!res.write(frame)) await once(res, 'drain', { signal });
}

createServer(async (req, res) => {
  const prompt =
    new URL(req.url ?? '/', 'http://localhost').searchParams.get('q') ??
    'Explain server-sent events in five lines.';

  const disconnect = new AbortController();
  res.on('close', () => {
    if (!res.writableEnded) disconnect.abort(); // client left early
  });
  res.writeHead(200, SSE_HEADERS);
  const heartbeat = setInterval(() => res.write(': ping\n\n'), 15_000);

  const result = streamAnswer(prompt, disconnect.signal);

  try {
    for await (const part of result.stream) {
      const frame = toSseEvent(part);
      if (frame && !disconnect.signal.aborted)
        await send(res, frame, disconnect.signal);
    }
  } catch (err) {
    if (!disconnect.signal.aborted) console.error('stream crashed', err);
  } finally {
    clearInterval(heartbeat);
    if (!res.writableEnded) res.end();
  }
}).listen(3001, () => console.log('SSE on http://localhost:3001'));

Three details carry the whole design:

• Disconnect detection uses res.on('close'), not req. The response's close event fires when the response completes or the connection drops early. Checking writableEnded tells the two apart: if we never called end(), the client left.
• send() respects backpressure. res.write() returns false when the buffer is full. The Node.js backpressure guide is clear that you should wait for drain before writing again. Because send() is awaited inside for await, the loop stops pulling from the model while the socket catches up. Passing { signal } to once means a disconnect during that wait rejects instead of hanging forever.
• The heartbeat is an SSE comment. Lines starting with : are ignored by SSE parsers, and the HTML spec suggests one about every 15 seconds because some proxies drop idle connections. It keeps the line alive while the model is thinking.

Run it with npx tsx server.ts and test with curl -N "http://localhost:3001/?q=Write+a+long+story". Press Ctrl+C mid-stream and the server logs provider call cancelled.

Here is the read loop with backpressure as a flow:

Diagram
100%
flowchart LR
  A["for await part of result.stream"] --> B["toSseEvent(part)"]
  B --> C{"res.write(frame)<br/>returned false?"}
  C -- "No" --> A
  C -- "Yes" --> D["await once(res, 'drain')"]
  D --> A
  E["res 'close' before end()"] --> F["disconnect.abort()"]
  F --> G["provider request cancelled"]
Zoom or open fullscreen for a closer look
visualized by

Step 4: a client that clicks "Stop"

// client.ts (run with: npx tsx client.ts)
const controller = new AbortController();
setTimeout(() => controller.abort(), 2_000); // simulate "Stop generating"

const res = await fetch('http://localhost:3001/?q=Write+a+long+story', {
  signal: controller.signal,
});

const decoder = new TextDecoder();
try {
  for await (const chunk of res.body!) {
    process.stdout.write(decoder.decode(chunk, { stream: true }));
  }
} catch (err) {
  if ((err as Error).name !== 'AbortError') throw err;
  console.log('\n[client] aborted');
}

After two seconds the client aborts, the socket closes, and the server's close handler cancels the model call. One signal, end to end.

Step 5: the same thing in a Next.js Route Handler

In the App Router you return a web Response, so disconnects arrive as req.signal and backpressure comes from ReadableStream's pull(), which is only called when the consumer wants more data.

// app/api/chat/route.ts
import { streamAnswer } from '@/lib/llm';
import { SSE_HEADERS, toSseEvent } from '@/lib/sse';

export async function POST(req: Request) {
  const { prompt } = (await req.json()) as { prompt: string };

  const cancelled = new AbortController();
  // Stop the provider call if the client disconnects OR the body is cancelled.
  const result = streamAnswer(
    prompt,
    AbortSignal.any([req.signal, cancelled.signal]),
  );

  const parts = result.stream[Symbol.asyncIterator]();
  const encoder = new TextEncoder();

  const body = new ReadableStream<Uint8Array>({
    // Called only when the consumer wants more: natural backpressure.
    async pull(controller) {
      while (true) {
        const { done, value } = await parts.next();
        if (done) return controller.close();
        const frame = toSseEvent(value);
        if (frame) return controller.enqueue(encoder.encode(frame));
      }
    },
    // Ending the iterator alone does not stop the provider request.
    cancel() {
      cancelled.abort();
    },
  });

  return new Response(body, { headers: SSE_HEADERS });
}

The while loop inside pull() matters: if a part maps to null, we keep reading until we have something to enqueue, so a pull never resolves empty-handed. If you deploy on Vercel, the AI SDK docs also note that function cancellation must be enabled with supportsCancellation in vercel.json for the abort to reach your function.

Common mistakes and their fixes

1. Not forwarding a signal to the model call. The browser stops, your server keeps streaming from the provider. Fix: always pass abortSignal, derived from res.on('close') or req.signal.
2. Assuming cancel() or breaking the loop stops the provider. In my testing with AI SDK 7, ending the iterator did not abort the underlying request; only the abort signal did. Fix: abort an AbortController in cancel() and combine it with AbortSignal.any.
3. Using textStream and wondering where errors went. The SDK's type docs say error parts are not surfaced in textStream. Fix: iterate result.stream and handle error and abort parts, or use onError.
4. Ignoring res.write()'s return value. On a slow mobile connection, frames pile up in memory. Fix: await drain when write() returns false.
5. A proxy buffering the whole response. Everything arrives at once at the end. Fix: send X-Accel-Buffering: no (nginx documents this header for disabling proxy buffering per response) and Cache-Control: no-transform.

Interview angle

For full-stack or AI-focused frontend roles, a common follow-up after "how does streaming work?" is "what happens when the user closes the tab?" or "how do you report an error after a 200?". A strong answer names the abort signal chain, backpressure, in-band error events and timeouts for stalled streams. Practise these in the streaming interview questions.

What's next

On Thursday, the case study How AI apps stream tokens from the model to the browser follows a token through the provider, your server, proxies and the browser. Next week we build the React side: a streaming chat UI without re-render jank.

Conclusion

Streaming is easy to start and easy to leak. Treat it like a subscription with cleanup: forward one abort signal from the browser to the model, pull from the model only as fast as the client reads, set first-chunk and between-chunk timeouts, and send errors as SSE events because the status code is already gone. Those four habits turn a demo endpoint into one you can put in front of real users.