LLM gateway: routing, fallbacks, caching and budgets
We'll cover the following
Visual summary

Download imageFree to share, please link back to this lesson.
An LLM gateway is one internal endpoint, usually OpenAI-compatible, that sits between every application and every model provider. Apps call the gateway with a virtual key. The gateway picks the provider, retries, caches, enforces budgets, applies guardrails and logs every call. LiteLLM proxy is a common open-source choice; Portkey, Kong AI Gateway and a homegrown service follow the same design.
Why a platform team builds one
- One API shape: switch provider by changing a model alias in config, not application code.
- Central keys: apps never hold provider keys. Each team gets a virtual key with its own allowed models, budget and rate limit.
- Reliability: retries and fallbacks when one provider throttles or goes down.
- Cost control: spend per team, caching, and routing easy work to cheaper models.
- One place for policy: guardrails, audit logs and PII rules apply to every app without each team rebuilding them.
The request path
app (virtual key)
-> authenticate key, check budget and rate limit
-> input guardrail (PII, injection)
-> cache lookup (exact or semantic)
-> router picks a deployment
-> provider call (retry, cooldown, fallback)
-> output guardrail
-> log tokens, cost, latency, model actually used
-> response (streamed or not)Routing, retries and fallbacks
| Mechanism | What it does | Use it for |
|---|---|---|
| Load balancing | Spreads one model alias over several deployments (regions, accounts) so one quota is not the ceiling | Throughput |
| Retry with backoff | Repeats on 429, 5xx and timeouts, honouring retry-after | Transient errors |
| Cooldown | Takes a deployment out of rotation for a short time after repeated failures | Not hammering a sick endpoint |
| Fallback | After retries are spent, sends the request to a different model alias | Provider outage or sustained throttling |
| Context-window fallback | Sends an over-long prompt to a larger-context model | Long inputs |
A config sketch (LiteLLM style)
model_list:
- model_name: chat-default # the alias apps use
litellm_params:
model: bedrock/<claude-model-id>
aws_region_name: us-east-1
- model_name: chat-default # same alias, second deployment: load balanced
litellm_params:
model: bedrock/<claude-model-id>
aws_region_name: us-west-2
- model_name: chat-backup # different provider for fallback
litellm_params:
model: openai/<model-id>
api_key: os.environ/OPENAI_API_KEY
router_settings:
routing_strategy: latency-based-routing
num_retries: 2
timeout: 30
allowed_fails: 3
cooldown_time: 30
fallbacks: [{"chat-default": ["chat-backup"]}]
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL # keys, teams and spend live hereField names change between versions, so treat this as the shape to remember and check the current docs when you write the real file.
Caching: three different things
| Cache | Key | Risk |
|---|---|---|
| Exact (gateway) | Hash of model, messages and parameters | Low hit rate unless prompts repeat (retries, eval runs, FAQs) |
| Semantic (gateway) | Embedding of the prompt, hit above a similarity threshold | A similar but different question gets the wrong stored answer; stale or cross-user answers |
| Provider prompt cache | Identical prompt prefix | None for correctness. The model still runs, only the prefix is cheaper |
- Use a high similarity threshold, a TTL, and a namespace per tenant so one customer never sees another's answer.
- Cache only read-only, low-risk traffic such as FAQs. Never cache answers that depend on user-specific tool calls or contain PII.
- Measure hit rate and wrong-hit rate. A cache that is fast and wrong is worse than none.
Cost tracking and FinOps
- Attribute every call: cost is tokens times the model's price. Tag each request with team, project and user (key, team id or metadata) so spend can be grouped.
- Budgets and alerts: a
max_budgetper key or team with a reset period, and an alert at about 80% so nobody is surprised. - Show-back vs charge-back: show teams their spend first, bill them later.
- Unit economics: report cost per resolved ticket or per document, not only dollars per month.
- Price table upkeep: custom, fine-tuned or provisioned models need prices entered by hand, and cached tokens are priced differently.
Levers, roughly in order of payoff: route easy requests to a smaller model, cache repeats, trim prompts, use provider prompt caching, send offline work through a batch API (about half price), set max_tokens, and cap agent loop iterations.
Guardrails at the gateway
Putting input checks (PII, prompt-injection classifiers) and output checks (toxicity, grounding) at the gateway gives every app the same protection. The costs are added latency, false positives that block real users, and one more dependency. Make policy configurable per team. Amazon Bedrock Guardrails can be called as one backend through its standalone ApplyGuardrail API.
How a gateway fails
- Single point of failure: run several stateless replicas behind a load balancer with health checks.
- Database in the hot path: writing spend logs synchronously slows every call. Batch or queue the writes.
- Streaming and fallback: once the first token has been sent you cannot switch models, so fallback only works before streaming starts.
- Retry storms: retries multiply load on a provider that is already throttling. Use jittered backoff and a retry budget.
- Config drift: keep the config in git and deploy it through CI like code.
Check yourselfA provider starts returning 429s at peak. The gateway retries each call three times and the incident gets worse. Why, and what do you change?
Retries multiply traffic against a provider that is already throttling, a retry storm. Use jittered exponential backoff with a retry budget, spread load over more deployments or regions, put the failing deployment in cooldown, and fall back to another model after retries instead of retrying forever.
Check yourselfWhy is a semantic cache riskier than an exact cache, and how do you make it safer?
It returns a stored answer for a prompt that is only similar, so a different question ("refund for order A" vs "order B") can get the wrong answer, and answers can be stale or leak across users. Use a high similarity threshold, TTLs, a cache namespace per tenant, only cache read-only low-risk traffic, and track the wrong-hit rate.
Summary: say it in the interview
- Retry the same deployment for transient errors, fall back to another model only when retries fail, and never retry a 400 or 401. A fallback changes quality, so log which model actually answered.
- "One OpenAI-compatible front door with virtual keys, per-team budgets, retries then fallbacks, a tenant-scoped cache and guardrails, and every call logged with the model that answered and its cost."