Latency vs Throughput

Why It Matters

Most performance issues come down to misunderstanding these two concepts.

Your system might be fast for a single user but fail under load. Or it might handle huge traffic but feel slow.

This is the difference between latency and throughput.

You will face this in:

• system design interviews
• scaling backend services
• optimizing APIs

What This Concept Actually Means

Latency

Time taken to complete a single request.

Example:

• API response time = 120ms

Throughput

Number of requests processed per unit time.

Example:

• System handles 10,000 requests per second (RPS)

Think like this:

• Latency = how fast one request is
• Throughput = how many requests you can handle

How It Works

Let’s take a video streaming service.

Scenario 1: Low Latency, Low Throughput

• Response time: 50ms
• Can handle: 100 requests/sec

Fast for users, but system crashes when traffic increases.

Scenario 2: High Throughput, High Latency

• Response time: 800ms
• Can handle: 50,000 requests/sec

System scales well but users feel delay.

Ideal Scenario

• Response time: <200ms
• Throughput: 20,000+ requests/sec

Balanced system.

Flow:

Diagram
flowchart LR
  User --> LoadBalancer
  LoadBalancer --> ServerCluster
  ServerCluster --> Database
visualized by

Key Techniques / Variations

1. Caching

• Reduces latency
• Example: cache reduces response from 300ms to 50ms

2. Horizontal Scaling

• Increases throughput
• Example: 1 server = 1k RPS, 10 servers = ~10k RPS

3. Asynchronous Processing

• Improves throughput
• Example: queue-based systems for background jobs

4. Database Optimization

• Indexing reduces latency
• Sharding improves throughput

Trade-offs and Design Decisions

When to prioritize latency

• real-time chat apps
• gaming systems

Target:

• <100ms response time

When to prioritize throughput

• analytics systems
• batch processing

Target:

• millions of events per minute

Pros of optimizing latency

• better user experience

Pros of optimizing throughput

• handles large scale traffic

Cons

• ultra-low latency systems are expensive
• high throughput systems can increase complexity

Architecture / Flow Diagram

Diagram
flowchart LR
  Client --> LoadBalancer
  LoadBalancer --> AppServers
  AppServers --> Cache
  AppServers --> Database
visualized by

Failure Modes To Watch

• High latency due to database bottlenecks
• Low throughput due to limited servers
• Network delays
• Poor caching strategy
• Blocking synchronous operations

Design Checklist

Ask yourself:

• What is acceptable latency? (e.g., <200ms)
• What throughput is required? (e.g., 10k RPS)
• Where is the bottleneck?
• Can I scale horizontally?
• Can I use caching or queues?

Summary

• Latency is time per request
• Throughput is requests per second
• Both are critical for system performance
• Optimizing one may impact the other
• Always design based on use case requirements