Mock design: an LLM gateway for 50 teams

Lesson 9.1 · 6 min read · by Vinay C

We'll cover the following

Visual summary

Infographic summarizing Mock design: an LLM gateway for 50 teams: Prompt: "Design an internal LLM gateway used by 50 engineering teams, with several model providers, per-team budgets…

Download imageFree to share, please link back to this lesson.

Prompt: "Design an internal LLM gateway used by 50 engineering teams, with several model providers, per-team budgets and an uptime target of 99.9%." Say the framework out loud: clarify, estimate, draw the design, go deep on two or three hard parts, then trade-offs.

1. Clarify

  • Who are the callers: services, notebooks, internal chat apps? Streaming needed? Which providers and models? Any data residency limits?
  • What must it enforce: authentication, per-team budgets, rate limits, guardrails, audit?
  • What does failure look like: is it fine to fail open on a policy check, and who gets paged?

2. Estimate

  • Assume 100 requests per second at peak, each about 1,800 tokens in and out and lasting 8 seconds.
  • Concurrency = rate x latency (Little's law): 100 x 8 = about 800 streams open at once. The gateway is I/O bound, so connection count and memory matter more than CPU.
  • Tokens: 100 x 60 x 1,800 is about 10.8 million tokens per minute. Provider quotas (tokens and requests per minute), not the gateway, are the real ceiling.
  • Spend logs: 100 writes per second to a database is fine if batched, but painful if synchronous.

3. High-level design

clients -> load balancer -> gateway replicas (stateless)
                              |-> Redis: rate-limit counters, cooldown state, cache
                              |-> Postgres: keys, teams, budgets, model aliases
                              |-> guardrail service (input and output)
                              |-> provider adapters -> Bedrock, OpenAI, others
                              |-> OTel export -> traces, metrics, spend log queue
config in git -> CI validates -> deploy (canary) ; spend queue -> batch writer -> Postgres

4. Deep dives

  • Rate limiting: a token bucket per key for requests per minute and tokens per minute in Redis. The input tokens are known up front, output tokens are estimated, and the counter is corrected after the call.
  • Budgets: a counter per team in Redis updated after each call and flushed to Postgres. Enforcement is slightly eventually consistent, so a team can overshoot by a small amount. Accept it, use a soft warning at 80% and a hard stop at 100%.
  • Reliability: retry with jitter, per-deployment cooldown, spread across regions, fallback to another provider before the first streamed token. Log the model that answered.
  • Isolation: cache keys include the team or tenant, so one team can never receive another's cached answer. Guardrail policy is configured per team.
  • Failure of dependencies: if Redis is down, fail open on rate limits (with a coarse local limit) but fail closed on authentication. If Postgres is down, keep serving from cached key and budget data and queue the spend writes.

5. Operations and rollout

  • SLOs: availability, p95 gateway overhead (target a few tens of milliseconds), and time to first token.
  • Config as code: model aliases, fallbacks and limits live in git, are validated in CI, and roll out by canary.
  • Observability: traces with tokens, cost, team and model, plus alerts on error rate, provider throttling and spend against budget.
  • Security: provider keys in a secrets manager, per-team virtual keys, audit log of admin changes.

6. Trade-offs to name

DecisionChoice and why
Build or adoptAdopt an open-source gateway such as LiteLLM and own the config, guardrails and operations. Build only what is unique to the company
Strict vs eventual budgetsEventual, because a hard real-time check on every call would put a database in the hot path
Semantic cacheOff by default, enabled per team for low-risk FAQs, because wrong hits are costly
Fail open or closedClosed for authentication and guardrails on sensitive data, open for soft limits
Check yourselfWhy is the gateway's CPU not the main sizing concern here?

Requests are I/O bound and long-lived. Concurrency is rate times latency, about 800 open streams in the example, so connection handling and memory matter, and the real ceiling is the provider token and request quotas.

Check yourselfA team exceeds its budget by a few percent before the gateway cuts them off. Is that a bug?

Not necessarily. Budget counters are updated after calls and flushed asynchronously, so enforcement is eventually consistent. That is a deliberate trade-off to keep the database out of the hot path. Use soft warnings, a hard stop, and small headroom, and say it out loud as a trade-off.

Summary: say it in the interview

  • "Stateless replicas behind a load balancer, Redis for hot counters and state, Postgres off the hot path, quotas as the real bottleneck, fallback before the first token, tenant-scoped caching, and everything configured in git."