Concept Walkthrough

How UTF-8 stores characters

UTF-8 is a variable-length encoding. ASCII characters like a or { take 1 byte. Characters from many other scripts take more: Devanagari letters such as न take 3 bytes, and most emoji take 4 bytes. A decoder needs all the bytes of a character before it can produce it.

Where the bug comes from

The network does not know about characters. TCP, proxies and the provider's server can deliver a stream in chunks of any size, so a chunk can end after the first byte of न and the next chunk starts with its other two bytes.

If you decode every chunk on its own, the decoder sees an incomplete sequence at the end of chunk one. It cannot wait, because you told it this is all the data, so it outputs the replacement character U+FFFD (shown as �). The leftover bytes at the start of chunk two are also invalid on their own, so they turn into replacement characters too. The original character is lost.

This explains the "works on my laptop" part. Locally the stream often arrives in different chunk sizes, and English-only testing uses 1-byte characters that can never be split. In production, with real Hindi traffic and different network paths, the split eventually happens.

The fix: a stateful decoder

TextDecoder can keep state between calls. Create one decoder for the whole stream and pass { stream: true } on every chunk. MDN describes this flag as indicating that more data will follow in later calls. The decoder then holds incomplete bytes internally and finishes the character on the next call. When the stream ends, call decoder.decode() with no arguments to flush.

TextDecoderStream does the same thing as a transform stream, which is why res.body.pipeThrough(new TextDecoderStream()) is safe.

Two layers of buffering

Keep two layers apart in your head:

1. Bytes to characters: handled by the stateful decoder.
2. Characters to messages (lines, SSE events, JSON objects): handled by your own string buffer.

A split character is a layer 1 problem. A split JSON event is a layer 2 problem. A correct client needs both.

One more detail: the decoder guarantees complete code points, not complete visible symbols. A family emoji or an emoji with a skin tone is several code points joined together, so it may briefly render as separate pieces until the next chunk arrives. That is normal streaming behaviour, not data loss.

Key Code Explained

// Reproduce the bug: split a 3-byte character across two chunks
const bytes = new TextEncoder().encode('नमस्ते');
const chunkA = bytes.slice(0, 2); // first 2 bytes of the 3-byte "न"
const chunkB = bytes.slice(2);

// Wrong: a fresh decode per chunk, no stream flag
const broken = new TextDecoder().decode(chunkA) + new TextDecoder().decode(chunkB);
console.log(broken.includes('�')); // true: replacement characters appeared

// Right: one decoder, stream: true, then a final flush
const decoder = new TextDecoder();
let fixed = decoder.decode(chunkA, { stream: true }); // holds the 2 bytes, returns ''
fixed += decoder.decode(chunkB, { stream: true });
fixed += decoder.decode(); // flush anything left
console.log(fixed === 'नमस्ते'); // true

// A test that tries every split point, so the bug cannot hide
import { describe, expect, it } from 'vitest';

function decodeChunks(chunks: Uint8Array[]): string {
  const decoder = new TextDecoder();
  let out = '';
  for (const c of chunks) out += decoder.decode(c, { stream: true });
  return out + decoder.decode();
}

describe('decodeChunks', () => {
  it('rebuilds text split at any byte', () => {
    const original = 'नमस्ते 👋 hello';
    const bytes = new TextEncoder().encode(original);
    for (let i = 0; i <= bytes.length; i++) {
      expect(decodeChunks([bytes.slice(0, i), bytes.slice(i)])).toBe(original);
    }
  });
});

The test is the useful habit here. Chunk sizes in production are out of your control, so instead of hoping for a lucky split, you test every possible split. If you ever swap the decoder for something else, the test catches a regression.

In a Node proxy, the cleanest fix is often to not decode at all: forward the raw bytes to the client and let the browser's decoder handle them. Decode on the server only when you need to inspect or transform the text.

Tradeoffs

What Interviewers Actually Check

• Whether you know UTF-8 is variable length and which characters use multiple bytes
• Whether you can explain where � comes from instead of guessing
• Whether you know the stream: true option and the final flush
• Whether you separate decoder state from message buffering
• Whether you would write a test that forces the split instead of relying on luck

Follow-Up Questions

1. Why does the same bug not appear with plain ASCII JSON payloads?
2. How would you detect this problem in production logs or monitoring?
3. What is the difference between a code point and a grapheme cluster, and why does it matter while streaming?
4. If your server transforms the text (for example, filtering words), where should decoding happen?
5. How would Intl.Segmenter help if you wanted to reveal text one visible character at a time?

Common Candidate Mistakes

• Blaming the model or the font for the � characters
• Writing chunk.toString() in a Node proxy and only testing in English
• Using the stream option but creating a new decoder on every read
• Removing � with a regex, which hides the bug and loses the real character
• Mixing up byte-level splits with event-level splits and fixing only one

Interview Readiness Checklist

Before you leave this question, make sure you can answer:

• Can you explain UTF-8 variable-length encoding in one or two sentences?
• Can you explain exactly how a split character turns into �?
• Can you write the correct decode loop with a flush?
• Can you explain why proxies can often skip decoding?
• Can you write a test that splits at every byte?

Summary

UTF-8 stores many characters in 2 to 4 bytes, and the network can split a stream in the middle of one. Decoding each chunk separately turns those incomplete bytes into U+FFFD, which is why Hindi text and emoji break while English looks fine. Use one TextDecoder with { stream: true } for every chunk and flush it at the end, or pipe through TextDecoderStream. Keep that separate from your line or event buffer, which handles messages split across chunks. In a proxy, forward bytes untouched when you do not need to read them, and test by splitting the input at every byte.