Architecture Walkthrough

Why AI routes need more than a normal limit

A normal API limit counts requests. An LLM route costs money per token, so a request limit alone leaves a gap: a user within the limit can still paste huge inputs and ask for long answers. Think of it like a mobile data plan: you limit both the number of calls and the data used.

Layer 1: who is calling

Use the authenticated user id. Fall back to IP only for anonymous users. Next.js 15 removed ip and geo from NextRequest because these values come from your host. On Vercel, the docs point to ipAddress from @vercel/functions. On other hosts, read the header your proxy sets, and only trust it if your proxy controls it.

Layer 2: request rate

Serverless instances do not share memory, so the counter must live in shared storage. The AI SDK docs show @upstash/ratelimit with @upstash/redis for this. Upstash supports fixedWindow, slidingWindow and tokenBucket. A sliding window avoids the burst you get at the edge of fixed windows. Call limit(identifier) before anything else. It returns success, limit, remaining and reset (a Unix time in milliseconds).

Layer 3: token budget

Keep a per-user, per-day token counter. Check it before calling the model, and add the real usage when the stream ends. Because you only know usage at the end, a user can go slightly over the budget on their last request. That is usually fine. If not, reserve an estimate up front.

Layer 4: bound a single request

Reject inputs above a size limit, limit how many messages you accept, and set maxOutputTokens. This puts a ceiling on the worst case for one request.

Layer 5: tell the client clearly

Return status 429 with a Retry-After header in seconds. In the UI, useChat exposes error and status. Show a calm message and disable sending. Do not auto-retry in a tight loop.

Key Code Explained

// app/api/chat/route.ts
import {
  streamText,
  convertToModelMessages,
  toUIMessageStream,
  createUIMessageStreamResponse,
  type UIMessage,
} from 'ai';
import { Ratelimit } from '@upstash/ratelimit';
import { Redis } from '@upstash/redis';
import { getSession } from '@/lib/auth'; // your own auth helper

const redis = Redis.fromEnv();

const perMinute = new Ratelimit({
  redis,
  limiter: Ratelimit.slidingWindow(10, '1 m'), // 10 requests per rolling minute
  prefix: 'chat-rpm',
});

const DAILY_TOKEN_BUDGET = 50_000; // pick from your own cost model
const MAX_INPUT_CHARS = 12_000;

export async function POST(req: Request) {
  const session = await getSession(req);
  if (!session) return new Response('Unauthorized', { status: 401 });
  const userId = session.userId;

  // 1. Request rate, checked before any model call
  const { success, reset } = await perMinute.limit(userId);
  if (!success) {
    const retryAfter = Math.max(1, Math.ceil((reset - Date.now()) / 1000));
    return new Response('Too many requests', {
      status: 429,
      headers: { 'Retry-After': String(retryAfter) },
    });
  }

  // 2. Daily token budget
  const day = new Date().toISOString().slice(0, 10);
  const budgetKey = `chat-tokens:${userId}:${day}`;
  const used = (await redis.get<number>(budgetKey)) ?? 0;
  if (used >= DAILY_TOKEN_BUDGET) {
    return new Response('Daily limit reached', { status: 429 });
  }

  // 3. Bound the size of this request
  const { messages }: { messages: UIMessage[] } = await req.json();
  if (JSON.stringify(messages).length > MAX_INPUT_CHARS) {
    return new Response('Message too long', { status: 413 });
  }

  const result = streamText({
    model: 'anthropic/claude-sonnet-5.5', // example id
    messages: await convertToModelMessages(messages),
    maxOutputTokens: 800,
    // 4. Record real usage when the stream ends
    onEnd: async ({ totalUsage }) => {
      await redis.incrby(budgetKey, totalUsage.totalTokens ?? 0);
      await redis.expire(budgetKey, 60 * 60 * 48);
    },
  });

  return createUIMessageStreamResponse({
    stream: toUIMessageStream({ stream: result.stream }),
  });
}

// In the chat Client Component (simplified)
const { error, status, clearError } = useChat();
const blocked = Boolean(error); // a 429 from the route surfaces here

return (
  <>
    {blocked && (
      <p role='status'>
        You have reached the limit. Please try again in a minute.{' '}
        <button onClick={() => clearError()}>OK</button>
      </p>
    )}
    <button type='submit' disabled={blocked || status !== 'ready'}>
      Send
    </button>
  </>
);

On Vercel Edge or Cloudflare Workers, the Upstash docs say to pass the pending promise from limit() to waitUntil so background work finishes. In a normal Node.js Route Handler you can usually ignore it.

Tradeoffs

What Interviewers Actually Check

• Whether you check limits before the model call, not after
• Whether you use shared storage instead of memory
• Whether you limit tokens and size, not only request count
• Whether you pick user id over IP and know req.ip is gone in Next.js 15
• Whether the client gets a 429 with Retry-After and a clear UI state

Follow-Up Questions

1. How would you give paid users higher limits without duplicating code?
2. A user opens five tabs and streams in all of them. How do you limit concurrent streams?
3. How would you reserve tokens up front so the budget is never exceeded?
4. What should happen to a stream that is already running when the budget runs out?
5. How do you rate-limit anonymous users fairly behind a shared college network?

Common Candidate Mistakes

• A Map counter in the route, which resets per instance
• Calling streamText before checking the limit, so blocked requests still cost money
• Only counting requests, while long inputs drain the budget
• Relying on req.ip, which no longer exists on NextRequest
• Retrying immediately on the client after a 429

Interview Readiness Checklist

Before you leave this question, make sure you can answer:

• Can you explain why serverless needs shared storage for limits?
• Can you write a sliding window check with @upstash/ratelimit?
• Can you add and update a daily token budget?
• Can you return 429 with a correct Retry-After value?
• Can you show a calm limit message in the chat UI?

Summary

Rate-limit an AI route in layers, all before the model call. Identify by user id, use a shared store like Redis for a sliding window request limit, keep a daily token budget that you update when the stream ends, and cap input size and maxOutputTokens. Return 429 with Retry-After so clients back off, and show a clear, calm message in the UI. Edge or firewall limits help against floods, but only your route can see who the user is and how many tokens they used.