Load Balancing

Why It Matters

As traffic grows, a single server cannot handle all requests.

Without load balancing:

• server crashes under high load
• users face downtime
• performance becomes unpredictable

Load balancing is one of the first steps toward building scalable systems.

You will face this in:

• system design interviews
• scaling backend APIs
• handling production traffic

What This Concept Actually Means

Load balancing distributes incoming requests across multiple servers.

Instead of:

• 1 server handling 10,000 requests per second

We use:

• 10 servers handling around 1,000 requests per second each

Think like this:

• Load balancer = traffic manager
• Servers = workers

How It Works

Let’s take a real-world API system.

Without Load Balancer

• 1 server capacity: 2,000 requests per second
• Incoming traffic: 8,000 requests per second

Result:

• server overload
• high latency (1,500 ms)
• failures

With Load Balancer

• 4 servers, each handling 2,000 requests per second
• Total capacity: around 8,000 requests per second

Result:

• stable latency (150 ms to 250 ms)
• no crashes

Flow:

Diagram
100%
flowchart LR
  User --> LoadBalancer
  LoadBalancer --> Server1
  LoadBalancer --> Server2
  LoadBalancer --> Server3
  LoadBalancer --> Server4
  Server1 --> Database
  Server2 --> Database
  Server3 --> Database
  Server4 --> Database
Zoom or open fullscreen for a closer look
visualized by

Key Techniques / Variations

1. Round Robin

Requests are distributed one by one across servers.

Example:

• Request 1 -> Server1
• Request 2 -> Server2

Simple but does not consider server load.

2. Least Connections

Send request to server with fewest active connections.

Better for uneven workloads.

3. Weighted Load Balancing

Assign weight based on server capacity.

Example:

• Server1 handles 2x traffic compared to Server2

4. Health Checks

Load balancer continuously checks server health.

• If server fails, traffic is rerouted

Trade-offs and Design Decisions

When to use load balancing

• traffic above single server capacity (e.g., above 2,000 RPS)
• need high availability

Pros

• improved scalability
• fault tolerance
• better performance

Cons

• added infrastructure cost
• extra network hop adds small latency (5 ms to 20 ms)

Real-world cost example

• Load balancer cost: 5k per month
• 4 servers cost: 40k per month
• Without load balancer, system fails at 2k RPS
• With load balancer, system handles 8k RPS reliably

Architecture / Flow Diagram

Diagram
100%
flowchart LR
  Client --> LoadBalancer
  LoadBalancer --> AppServer1
  LoadBalancer --> AppServer2
  AppServer1 --> Database
  AppServer2 --> Database
Zoom or open fullscreen for a closer look
visualized by

Failure Modes To Watch

• load balancer becomes single point of failure
• uneven traffic distribution
• slow health checks
• database bottleneck even after scaling servers

Design Checklist

Ask yourself:

• What is current traffic? (e.g., 5,000 RPS)
• What is per server capacity? (e.g., 1,500 RPS)
• How many servers do I need?
• What load balancing strategy fits best?
• Do I have health checks in place?

Summary

• Load balancing distributes traffic across servers
• It prevents overload and improves availability
• Essential for scaling beyond a single machine
• Different strategies exist based on system needs
• Always combine with monitoring and health checks