Rate Limiting LLM API Calls Without Killing Developer Experience

50 parallel requests at midnight can push LLM provider spend past your plan before anyone notices. This often starts with an unbounded demo, a helpful integration test, or one API consumer. When monitoring finally alerts you, the bill has already arrived.

LLM API limits differ from limits on a predictable endpoint. A REST route may take 5ms and return 200 bytes. An LLM call may run for 2 to 30 seconds and use 200 to 8,000 tokens. For cost control, token usage matters more than request count, so frequency-only limits leave the main exposure uncovered.

Why One Rate Limit Is Never Enough

One limit always leaves a gap. A global request ceiling can block abuse while letting one API key consume capacity needed by everyone else. A per-key token budget does not prevent concurrent bursts from filling the backend connection pool. Layered controls cover different failure modes.

The five gateway controls are a global model cap, per-API-key token budget, per-tenant quota, concurrency limit, and sliding-window burst limit. Each catches failures the others can miss.

Global Model Cap

A global model cap sets the financial ceiling. Across every caller, API key, and tenant, usage stays below X tokens per minute or per day for one model. Once reached, new requests for that model fail until the window resets.

In a routes config, a global cap policy looks like this:

{
  "name": "rate-limit-global-gpt4o",
  "policy": "rate-limit-inbound",
  "config": {
    "rateLimitBy": "global",
    "tokenBudget": 500000,
    "windowSeconds": 60,
    "model": "gpt-4o"
  }
}

That cap is not enough by itself. One API key using 490,000 of 500,000 tokens can leave every other consumer without capacity. Per-key budgets provide the missing boundary.

Per-API-Key Token Budget

Each API key represents a caller: a developer, an integration, a product team. Attaching a token budget to the key enforces fairness and attributes cost to the right account for billing.

Use daily or monthly per-key limits instead of per-minute limits. A per-minute setting either trips during ordinary bursts or is made too large to protect anything. Daily and monthly resets fit both LLM cost accumulation and billing statements.

After a per-key budget runs out, return 429 with the limit that fired and its reset time. Opaque rate limits are a frequent cause of developer support requests.

Per-Tenant Quota

A multi-tenant product with several services and integrations per organization needs an aggregate quota above the key layer. An organization might create 10 API keys for 10 services, each with its own budget, while the organization quota remains their shared ceiling.

This fits billing tiers directly. A starter organization gets 100K tokens per month across all keys, while larger plans receive more. The gateway, rather than application code, enforces the quota consistently for every service in the organization.

Concurrency Limit

Token budgets and request counters do not stop a thundering herd. One consumer making 200 parallel requests may stay under its token budget initially, yet fill the connection pool and increase latency for every other in-flight request.

Concurrency limits cap in-flight requests per key at any moment. Once the cap is reached, new requests wait or receive 429 with a short Retry-After. Because LLM responses take seconds, concurrency often matters more to backend stability than token budgets.

Sliding-Window Burst Limit

A sliding window controls brief spikes. A consumer may make 30 requests per minute, but cannot place all 30 in the first 5 seconds and then restart the counter. The window measures request distribution across the previous 60 seconds at every point.

Interactive applications meet the burst limit first. Make it too strict and normal behavior, such as clicking regenerate several times quickly, fails. Make it too loose and scripted abuse passes through. Traffic shape should determine the value, so keep it in config instead of compiled code.

Surfacing Rejection Reasons That Actually Help

Many gateway setups mishandle the 429 body. An empty response, or one containing only "too many requests," gives the consumer no next step. They cannot tell which of the five limits fired, whether it is temporary or final for the day, or when retrying makes sense.

A helpful 429 body names the limit, reset timestamp, and retry window:

{
  "error": "rate_limit_exceeded",
  "limit_type": "per_key_daily_token_budget",
  "key_id": "zpl_live_k8a3x9",
  "budget_used": 100000,
  "budget_limit": 100000,
  "reset_at": "2026-08-09T00:00:00Z",
  "retry_after_seconds": 56340
}

The Retry-After HTTP header should repeat that window. Good API clients use it to back off automatically. It also distinguishes a temporary per-minute burst rejection, retry in 30 seconds, from a daily budget exhaustion, return tomorrow. Those cases need different client handling.

What Gateway Rate Limiting Does Not Cover

Gateway rate limiting does not replace product cost controls or fraud detection. If your product bills by token, its billing system must independently record actual usage. The gateway applies hard ceilings, while billing keeps invoices accurate.

Rate limits also cannot stop a compromised API key from spending its budget before rotation. Key rotation and anomaly detection are separate jobs. These five controls cover runaway spend and consumer fairness, providing a foundation for additional layers rather than replacing them.

Keeping the Config Reviewable

Putting all five controls in route config instead of application code makes changes auditable. A PR changing a token budget is visible to the team, as is one adding a per-org quota for an integration. If a limit is too strict, revert it in git instead of deploying a hotfix.

{
  "route": "POST /api/chat",
  "policies": [
    { "name": "api-key-auth", "policy": "api-key-inbound" },
    {
      "name": "burst-limit",
      "policy": "rate-limit-inbound",
      "config": {
        "rateLimitBy": "apiKey",
        "requestsPerWindow": 30,
        "windowSeconds": 60
      }
    },
    {
      "name": "daily-token-budget",
      "policy": "token-budget-inbound",
      "config": {
        "rateLimitBy": "apiKey",
        "tokenBudget": 100000,
        "windowSeconds": 86400
      }
    }
  ]
}

Policies run in sequence. Auth comes first, identifying the key before limits are checked. The burst window precedes the daily budget because burst rejection is temporary, while budget exhaustion lasts for the day. The sequence exposes rejection priority through limit_type in the 429 response: consumers first see request rate limiting, then token limiting once that clears. Both signals support better client code.