Rate limiting, backpressure and load shedding
We'll cover the following
Why
- Protect against overload and attacks (DoS).
- Fairness: one noisy customer cannot starve others.
- Cost control for expensive calls (LLM, SMS APIs).
- Abuse prevention: scraping, brute-forcing logins.
- Business tiers: free 100 req/min, paid 10,000.
Algorithms
| Algorithm | How | Note |
|---|---|---|
| Fixed window | Count per minute, reset at :00 | Simple, but boundary spike: 100 at 0:59 plus 100 at 1:01 doubles the rate |
| Sliding window | Count requests in the last N seconds | Accurate, costs more memory |
| Token bucket | Bucket refills at a steady rate (e.g. 10 tokens/sec) up to a max size; each request takes a token; empty means reject | Allows short bursts with a steady average. Most common answer. |
| Leaky bucket | Requests drain at a constant rate | Smooths traffic, no bursts |
Correction: in a token bucket tokens are not returned when a request completes; they refill over time. A scheme where the token comes back on completion is a concurrency limiter (max N in-flight requests), valid but different.
Where, what to key on, what to return
- Gateway by default (cheap, central). Add a second layer on expensive endpoints inside services as defense in depth. Client-side limiting is politeness only.
- Key on user ID, API key or IP.
- Reject with HTTP 429 and a
Retry-Afterheader.
Distributed problem: 50 gateways, 100 req/min per user
With only local counters, requests spread across gateways so no one knows the real count. Options:
- Shared counter in Redis (the standard). Use an atomic operation:
INCR user:42:minuteplusEXPIRE, or a Lua script, so gateways cannot race. Add replicas. Costs ~1ms hop per request. Decide fail open (let traffic through; usual for general APIs) vs fail closed (block; for sensitive things like login attempts) if Redis dies. - Sticky routing by user. Fast, local counters. Downsides: uneven load, counters lost when a gateway dies, reshuffle when gateways change (consistent hashing limits it to 1/N).
- Approximate local counters that sync every second or so. Slight overshoot (105 instead of 100) but no Redis call on the hot path; for massive scale.
Backpressure and load shedding: when everyone together is too much
Rate limiting caps what one caller may send. Backpressure and load shedding decide what the system does when total demand is above capacity, even though no single caller broke a limit (a viral post, a retry storm, a slow dependency).
| Technique | How | Note |
|---|---|---|
| Bounded queues + backpressure | Give every queue a maximum size. When it is full, tell the producer to slow down: block it, reject with an error, or let the consumer pull at its own pace (Kafka consumers pull, TCP uses a receive window). | An unbounded queue hides the overload, then turns it into huge latency and out-of-memory crashes. |
| Load shedding | When saturated, reject work early and cheaply: return 503 + Retry-After. Drop the least important first (recommendations before checkout; background sync before user clicks). | Failing 20% fast beats making 100% of requests slow. |
| Timeouts and deadlines | Every call has a timeout, and the remaining deadline is passed downstream so a service never works on a request whose caller already gave up. | Without them, threads pile up waiting on a slow dependency and the failure spreads upstream. |
| Retries with backoff and jitter | Wait 1s, 2s, 4s... plus random jitter, with a cap and a retry budget (e.g. at most 10% extra traffic from retries). | Naive retries multiply load exactly when the system is weakest (retry storm). Retrying only makes sense for idempotent calls. |
| Concurrency limits | Cap in-flight requests per instance (a concurrency limiter, see the token bucket correction above); adaptive versions lower the limit when latency rises. | Protects a service from itself when a downstream slows down. |
Why "just queue it all" fails
- Latency = queue length / service rate. A long queue means every request waits past its client timeout, so the system is busy but nobody gets a useful answer.
- Clients time out and retry, adding more load. The backlog can keep the system down even after traffic returns to normal (a metastable failure).
- Queues are right for work that can wait (emails, reports). They are wrong in front of a synchronous user request.
Check yourselfTraffic is 3x capacity for ten minutes. Do you queue everything or reject some?
Reject some. Shed the lowest-priority requests early with 503 + Retry-After, keep a small bounded queue for the rest, and have clients back off with jitter. An unbounded queue turns overload into latency beyond every timeout, so you serve nobody, and the backlog keeps the system down after the spike ends.
Summary: say it in the interview
- "Redis with atomic increments, accepting a 1 ms hop. If that becomes a bottleneck, local counters with periodic sync and accept slight overshoot."
- "Rate limiting protects me from one caller; backpressure and load shedding protect me from all callers at once. I would rather fail 20% fast than 100% slowly."