LLM gateway: routing, fallbacks, caching and budgets

Lesson 5.1 · 8 min read · by Vinay C

We'll cover the following

Visual summary

Infographic summarizing LLM gateway: routing, fallbacks, caching and budgets: An LLM gateway is one internal endpoint, usually OpenAI-compatible, that sits between every application and every…

Download imageFree to share, please link back to this lesson.

An LLM gateway is one internal endpoint, usually OpenAI-compatible, that sits between every application and every model provider. Apps call the gateway with a virtual key. The gateway picks the provider, retries, caches, enforces budgets, applies guardrails and logs every call. LiteLLM proxy is a common open-source choice; Portkey, Kong AI Gateway and a homegrown service follow the same design.

Why a platform team builds one

  • One API shape: switch provider by changing a model alias in config, not application code.
  • Central keys: apps never hold provider keys. Each team gets a virtual key with its own allowed models, budget and rate limit.
  • Reliability: retries and fallbacks when one provider throttles or goes down.
  • Cost control: spend per team, caching, and routing easy work to cheaper models.
  • One place for policy: guardrails, audit logs and PII rules apply to every app without each team rebuilding them.

The request path

app (virtual key)
  -> authenticate key, check budget and rate limit
  -> input guardrail (PII, injection)
  -> cache lookup (exact or semantic)
  -> router picks a deployment
  -> provider call (retry, cooldown, fallback)
  -> output guardrail
  -> log tokens, cost, latency, model actually used
  -> response (streamed or not)

Routing, retries and fallbacks

MechanismWhat it doesUse it for
Load balancingSpreads one model alias over several deployments (regions, accounts) so one quota is not the ceilingThroughput
Retry with backoffRepeats on 429, 5xx and timeouts, honouring retry-afterTransient errors
CooldownTakes a deployment out of rotation for a short time after repeated failuresNot hammering a sick endpoint
FallbackAfter retries are spent, sends the request to a different model aliasProvider outage or sustained throttling
Context-window fallbackSends an over-long prompt to a larger-context modelLong inputs

A config sketch (LiteLLM style)

model_list:
  - model_name: chat-default          # the alias apps use
    litellm_params:
      model: bedrock/<claude-model-id>
      aws_region_name: us-east-1
  - model_name: chat-default          # same alias, second deployment: load balanced
    litellm_params:
      model: bedrock/<claude-model-id>
      aws_region_name: us-west-2
  - model_name: chat-backup           # different provider for fallback
    litellm_params:
      model: openai/<model-id>
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  routing_strategy: latency-based-routing
  num_retries: 2
  timeout: 30
  allowed_fails: 3
  cooldown_time: 30
  fallbacks: [{"chat-default": ["chat-backup"]}]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL   # keys, teams and spend live here

Field names change between versions, so treat this as the shape to remember and check the current docs when you write the real file.

Caching: three different things

CacheKeyRisk
Exact (gateway)Hash of model, messages and parametersLow hit rate unless prompts repeat (retries, eval runs, FAQs)
Semantic (gateway)Embedding of the prompt, hit above a similarity thresholdA similar but different question gets the wrong stored answer; stale or cross-user answers
Provider prompt cacheIdentical prompt prefixNone for correctness. The model still runs, only the prefix is cheaper
  • Use a high similarity threshold, a TTL, and a namespace per tenant so one customer never sees another's answer.
  • Cache only read-only, low-risk traffic such as FAQs. Never cache answers that depend on user-specific tool calls or contain PII.
  • Measure hit rate and wrong-hit rate. A cache that is fast and wrong is worse than none.

Cost tracking and FinOps

  • Attribute every call: cost is tokens times the model's price. Tag each request with team, project and user (key, team id or metadata) so spend can be grouped.
  • Budgets and alerts: a max_budget per key or team with a reset period, and an alert at about 80% so nobody is surprised.
  • Show-back vs charge-back: show teams their spend first, bill them later.
  • Unit economics: report cost per resolved ticket or per document, not only dollars per month.
  • Price table upkeep: custom, fine-tuned or provisioned models need prices entered by hand, and cached tokens are priced differently.

Levers, roughly in order of payoff: route easy requests to a smaller model, cache repeats, trim prompts, use provider prompt caching, send offline work through a batch API (about half price), set max_tokens, and cap agent loop iterations.

Guardrails at the gateway

Putting input checks (PII, prompt-injection classifiers) and output checks (toxicity, grounding) at the gateway gives every app the same protection. The costs are added latency, false positives that block real users, and one more dependency. Make policy configurable per team. Amazon Bedrock Guardrails can be called as one backend through its standalone ApplyGuardrail API.

How a gateway fails

  • Single point of failure: run several stateless replicas behind a load balancer with health checks.
  • Database in the hot path: writing spend logs synchronously slows every call. Batch or queue the writes.
  • Streaming and fallback: once the first token has been sent you cannot switch models, so fallback only works before streaming starts.
  • Retry storms: retries multiply load on a provider that is already throttling. Use jittered backoff and a retry budget.
  • Config drift: keep the config in git and deploy it through CI like code.
Check yourselfA provider starts returning 429s at peak. The gateway retries each call three times and the incident gets worse. Why, and what do you change?

Retries multiply traffic against a provider that is already throttling, a retry storm. Use jittered exponential backoff with a retry budget, spread load over more deployments or regions, put the failing deployment in cooldown, and fall back to another model after retries instead of retrying forever.

Check yourselfWhy is a semantic cache riskier than an exact cache, and how do you make it safer?

It returns a stored answer for a prompt that is only similar, so a different question ("refund for order A" vs "order B") can get the wrong answer, and answers can be stale or leak across users. Use a high similarity threshold, TTLs, a cache namespace per tenant, only cache read-only low-risk traffic, and track the wrong-hit rate.

Summary: say it in the interview

  • Retry the same deployment for transient errors, fall back to another model only when retries fail, and never retry a 400 or 401. A fallback changes quality, so log which model actually answered.
  • "One OpenAI-compatible front door with virtual keys, per-team budgets, retries then fallbacks, a tenant-scoped cache and guardrails, and every call logged with the model that answered and its cost."