Load balancing and API gateways

A load balancer (LB) sits in front of many identical servers and spreads requests. It gives scale and survives a server dying because it health-checks servers and stops sending traffic to dead ones.

Big systems often use both: L4 at the edge for raw throughput feeding a fleet of L7 LBs. For most interview designs say "L7 LB" and move on; mention L4 for non-HTTP traffic or raw throughput.

• Round-robin: take turns; fine when servers and requests are equal.
• Least connections: fewest active requests; better when request times vary.
• Weighted: bigger servers get more traffic.
• IP hash / consistent hashing: same client or key goes to the same server (sticky sessions, per-server caches).

If a session lives in server 1 memory, requests landing on servers 2 or 3 break (random logouts). Fix: externalize the session into Redis (best), carry state in a token (JWT), or use sticky sessions as a fragile last resort (server 1 dies, those sessions die). Any server must be able to handle any request, which is why LBs work at all, and why gateway instances must be stateless too.

• Run several LBs: active-passive with a virtual IP that a standby takes over (keepalived/VRRP), or active-active with DNS returning multiple LB IPs.
• Use a managed LB (ALB/NLB, GCP LB): already a distributed fleet, multi-AZ.
• Anycast for global setups: same IP announced from many places, traffic goes to the nearest healthy one.

• LB: spreads traffic across identical servers (availability and distribution).
• API gateway: the front door for APIs. Auth, rate limiting, API keys, routing to different microservices, request transformation, versioning. Under the hood it is an L7 proxy.

Client -> LB #1 -> [Gateway x N] -> Service A (internal LB or discovery -> instances)
                                      -> Service B (internal LB or discovery -> instances)

• LB #1 balances across the gateway instances (the gateway is also a stateless fleet).
• Routing (the gateway always does this) decides which service from the path: "which kitchen, pizza or sushi?".
• Balancing decides which instance of that service: "which chef in that kitchen?".

In Kubernetes the gateway calls http://service-a and K8s DNS plus kube-proxy spread over the pods (Option 1 built in). Scaling A from 100 to 200 instances is invisible to the gateway.

• The source IP is part of every connection; you cannot hide it.
• L4 passthrough: the backend sees the client IP directly. L7 proxy: the backend sees the LB IP, so the LB adds an X-Forwarded-For header.
• Many users behind one office or carrier (NAT) share an IP, so use auth tokens to identify people.