Architecture Walkthrough

The goal

The data path is browser → your server → LLM provider, and tokens flow back the same way. "Without buffering" means every chunk the provider sends should leave your server as soon as it arrives, not sit in memory until the full answer is ready. Think of your server as a pipe, not a bucket.

Step 1: never read the whole body

fetch in Node returns as soon as headers arrive, and upstream.body is a web ReadableStream. The common mistake is const data = await upstream.text() followed by res.send(data). That turns the pipe into a bucket. Instead, pass the stream itself along.

• Next.js Route Handler: return new Response(upstream.body, { headers }). The Next.js docs show Route Handlers returning a ReadableStream in a Response for streaming.
• Express or plain Node: res is a Node writable stream, not a web stream. Convert with Readable.fromWeb(upstream.body) and connect them with pipeline(), which also handles backpressure and errors.

Step 2: remove hidden buffers

Your code can be perfect and the response still arrives all at once, because something in between collects it:

• Compression middleware. The Express compression README says it does not work out of the box with server-sent events, because compression collects data before sending. Skip it for streaming routes or call res.flush().
• Reverse proxies. nginx buffers proxied responses by default when proxy_buffering is on. An upstream can turn this off per response by sending X-Accel-Buffering: no.
• Caches and CDNs. Send Cache-Control: no-cache and the right Content-Type (for example text/event-stream), so the response is treated as live data.

Step 3: handle disconnects

If the user closes the tab, your server should abort the upstream request. Otherwise the model keeps generating output tokens you pay for. In Node, the response close event fires when the connection ends, so abort an AbortController there. In a Route Handler, pass req.signal to the upstream fetch and test that your host actually fires it.

Optional: observe or reshape

To hide the provider's event format or log usage, put a TransformStream in the middle with pipeThrough(). Each chunk is processed and passed on immediately, so streaming is preserved.

Key Code Explained

// app/api/chat/route.ts (Next.js Route Handler)
export async function POST(req: Request) {
  const { messages } = (await req.json()) as { messages: unknown };
  // validate messages, check auth and rate limits here

  const upstream = await fetch(process.env.LLM_STREAM_URL!, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${process.env.LLM_API_KEY}`, // never reaches the browser
    },
    body: JSON.stringify({ messages, stream: true }),
    signal: req.signal, // abort upstream if the client disconnects
  });

  if (!upstream.ok || !upstream.body) {
    // Do not forward provider error details to the client
    return Response.json({ error: 'Upstream error' }, { status: 502 });
  }

  // Hand the stream straight through: no await upstream.text()
  return new Response(upstream.body, {
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'X-Accel-Buffering': 'no', // ask nginx not to buffer this response
    },
  });
}

// Express version
import express from 'express';
import { Readable } from 'node:stream';
import { pipeline } from 'node:stream/promises';

const app = express(); // note: no compression() on this route
app.use(express.json());

app.post('/api/chat', async (req, res) => {
  const controller = new AbortController();
  // 'close' fires when the response finishes or the client disconnects
  res.on('close', () => {
    if (!res.writableFinished) controller.abort();
  });

  try {
    const upstream = await fetch(process.env.LLM_STREAM_URL!, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: `Bearer ${process.env.LLM_API_KEY}`,
      },
      body: JSON.stringify({ messages: req.body.messages, stream: true }),
      signal: controller.signal,
    });

    if (!upstream.ok || !upstream.body) {
      res.status(502).json({ error: 'Upstream error' });
      return;
    }

    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'X-Accel-Buffering': 'no',
    });

    // Web stream -> Node stream -> response, with backpressure and error handling
    await pipeline(Readable.fromWeb(upstream.body), res);
  } catch (err) {
    if (controller.signal.aborted) return; // client left, nothing to send
    if (!res.headersSent) res.status(500).json({ error: 'Stream failed' });
    else res.destroy();
  }
});

Readable.fromWeb bridges the web stream from fetch into a Node stream, and pipeline connects it to res. Both versions forward raw bytes without decoding, which also avoids the split multi-byte character problem on the server.

Tradeoffs

What Interviewers Actually Check

• Whether you pass the stream through instead of reading it into memory
• Whether you know how to bridge web streams and Node streams
• Whether you can name the real-world buffering culprits (compression, nginx, caches)
• Whether you abort the upstream call on client disconnect
• Whether you keep the API key and provider error details on the server

Follow-Up Questions

1. How would you count output tokens for billing while still streaming?
2. How would you use ReadableStream.tee(), and what is the memory risk if one branch is slow?
3. How would you add per-user rate limiting before the upstream call?
4. What changes if the provider returns an error event halfway through the stream?
5. How would you test that the proxy really streams, rather than only checking the final output?

Common Candidate Mistakes

• Using await upstream.json() because that is how every other API route is written
• Keeping app.use(compression()) globally and not noticing it affects the stream
• Testing only on localhost, where no reverse proxy buffers the response
• Letting the upstream request continue after the client has gone
• Returning the provider's raw error JSON, which can reveal internal configuration

Interview Readiness Checklist

Before you leave this question, make sure you can answer:

• Can you write a streaming proxy in a Route Handler from memory?
• Can you write the Express version with Readable.fromWeb and pipeline?
• Can you list three things that can buffer a streaming response?
• Can you explain how a client disconnect reaches the provider?
• Can you add a transform step without losing incremental delivery?

Summary

A streaming proxy is a pipe, not a bucket. Start the upstream fetch, check the status, and hand upstream.body to your response: new Response(upstream.body) in a Next.js Route Handler, or pipeline(Readable.fromWeb(upstream.body), res) in Express. Then remove the layers that buffer silently: compression middleware, nginx proxy buffering (send X-Accel-Buffering: no) and caches (send Cache-Control: no-cache). Abort the upstream request when the client disconnects so you stop paying for unseen tokens, and add a TransformStream only when you need to reshape or observe the events.