Mock design: an LLM gateway for 50 teams
We'll cover the following
Visual summary

Download imageFree to share, please link back to this lesson.
Prompt: "Design an internal LLM gateway used by 50 engineering teams, with several model providers, per-team budgets and an uptime target of 99.9%." Say the framework out loud: clarify, estimate, draw the design, go deep on two or three hard parts, then trade-offs.
1. Clarify
- Who are the callers: services, notebooks, internal chat apps? Streaming needed? Which providers and models? Any data residency limits?
- What must it enforce: authentication, per-team budgets, rate limits, guardrails, audit?
- What does failure look like: is it fine to fail open on a policy check, and who gets paged?
2. Estimate
- Assume 100 requests per second at peak, each about 1,800 tokens in and out and lasting 8 seconds.
- Concurrency = rate x latency (Little's law): 100 x 8 = about 800 streams open at once. The gateway is I/O bound, so connection count and memory matter more than CPU.
- Tokens: 100 x 60 x 1,800 is about 10.8 million tokens per minute. Provider quotas (tokens and requests per minute), not the gateway, are the real ceiling.
- Spend logs: 100 writes per second to a database is fine if batched, but painful if synchronous.
3. High-level design
clients -> load balancer -> gateway replicas (stateless)
|-> Redis: rate-limit counters, cooldown state, cache
|-> Postgres: keys, teams, budgets, model aliases
|-> guardrail service (input and output)
|-> provider adapters -> Bedrock, OpenAI, others
|-> OTel export -> traces, metrics, spend log queue
config in git -> CI validates -> deploy (canary) ; spend queue -> batch writer -> Postgres4. Deep dives
- Rate limiting: a token bucket per key for requests per minute and tokens per minute in Redis. The input tokens are known up front, output tokens are estimated, and the counter is corrected after the call.
- Budgets: a counter per team in Redis updated after each call and flushed to Postgres. Enforcement is slightly eventually consistent, so a team can overshoot by a small amount. Accept it, use a soft warning at 80% and a hard stop at 100%.
- Reliability: retry with jitter, per-deployment cooldown, spread across regions, fallback to another provider before the first streamed token. Log the model that answered.
- Isolation: cache keys include the team or tenant, so one team can never receive another's cached answer. Guardrail policy is configured per team.
- Failure of dependencies: if Redis is down, fail open on rate limits (with a coarse local limit) but fail closed on authentication. If Postgres is down, keep serving from cached key and budget data and queue the spend writes.
5. Operations and rollout
- SLOs: availability, p95 gateway overhead (target a few tens of milliseconds), and time to first token.
- Config as code: model aliases, fallbacks and limits live in git, are validated in CI, and roll out by canary.
- Observability: traces with tokens, cost, team and model, plus alerts on error rate, provider throttling and spend against budget.
- Security: provider keys in a secrets manager, per-team virtual keys, audit log of admin changes.
6. Trade-offs to name
| Decision | Choice and why |
|---|---|
| Build or adopt | Adopt an open-source gateway such as LiteLLM and own the config, guardrails and operations. Build only what is unique to the company |
| Strict vs eventual budgets | Eventual, because a hard real-time check on every call would put a database in the hot path |
| Semantic cache | Off by default, enabled per team for low-risk FAQs, because wrong hits are costly |
| Fail open or closed | Closed for authentication and guardrails on sensitive data, open for soft limits |
Check yourselfWhy is the gateway's CPU not the main sizing concern here?
Requests are I/O bound and long-lived. Concurrency is rate times latency, about 800 open streams in the example, so connection handling and memory matter, and the real ceiling is the provider token and request quotas.
Check yourselfA team exceeds its budget by a few percent before the gateway cuts them off. Is that a bug?
Not necessarily. Budget counters are updated after calls and flushed asynchronously, so enforcement is eventually consistent. That is a deliberate trade-off to keep the database out of the hot path. Use soft warnings, a hard stop, and small headroom, and say it out loud as a trade-off.
Summary: say it in the interview
- "Stateless replicas behind a load balancer, Redis for hot counters and state, Postgres off the hot path, quotas as the real bottleneck, fallback before the first token, tenant-scoped caching, and everything configured in git."