Standard request limits miss token volume, which this article adds as a gateway control. A team in our early access beta needed three weeks to spot that one integration partner's nightly batch job used the entire monthly token allocation in a single night. The job sent only a few hundred requests, so request limits missed it. Each request was simply very large. The billing alert arrived after the damage.
Request count is a poor stand-in for LLM cost. Token count is the useful measure. That changes gateway rate-limiting design.
Why Request Limits Alone Fall Short
Traditional API limiting asks how many requests a key sent in the last minute. LLM billing instead charges input and output tokens, and both can differ by orders of magnitude between requests.
A small chat completion may use 200 input tokens and return 150 output tokens. A document summary may use 12,000 input tokens and return 800 output tokens. Both count as one request. "100 requests per minute" treats them alike, although one costs about 40x more.
If downstream consumers pay for token usage, or several teams and applications share an LLM budget, the gateway needs to meter tokens rather than requests.
Where Token Counts Are Available
The OpenAI API (and compatible APIs from Anthropic, Cohere, and most providers) returns token usage in every response:
{
"id": "chatcmpl-abc123",
"object": "chat.completion",
"usage": {
"prompt_tokens": 1847,
"completion_tokens": 312,
"total_tokens": 2159
}
}The gateway sees this response on its way to the caller, making the outbound path the natural metering point: read usage.total_tokens, apply it to the key's running budget, and reject later requests after exhaustion.
Streaming responses add one complication. With SSE, the usage block usually arrives after all content chunks. The gateway must buffer or inspect the final chunk for the token count. Zuplo's token budget policy reads the trailing [DONE] event and final delta chunk containing the usage summary.
Budget Scope Choices
Choose the budget scope based on what you need to protect. In production, we use three levels:
Per-key (per-consumer) budget
Give each API key a token budget and reset window. Once it is spent, the key receives 429 responses until reset. This fits protection against one consumer exhausting shared upstream capacity.
{
"name": "token-budget-per-key",
"handler": "token-budget-inbound",
"options": {
"key": "api-key",
"budget_tokens": 500000,
"window": "1d",
"count_field": "usage.total_tokens",
"on_exhausted": "reject"
}
}Per-application budget
Traffic through one route or route group uses a shared budget. This suits one logical application with multiple keys, or a cap on spending for a model or feature regardless of caller.
Per-user (derived from JWT claims)
When API keys belong to organizations but users have JWT tokens, derive the budget identity from the token. You can then provide per-seat or per-user billing over a shared upstream LLM account.
Reset Window Semantics
Billing alignment matters here. Budget windows should fit the billing cycle, and consumers need predictable rules for when a window resets.
Options:
- Calendar day (UTC midnight reset): Easy to explain and predictable. Developers receive N tokens daily, with the counter resetting at midnight UTC. Good for developer tier quotas.
- Rolling 30 days from key creation: Matches monthly customer billing. Each key follows its activation date, avoiding one shared reset spike.
- Monthly calendar (first of month UTC): Straightforward and compatible with financial reporting. Many simultaneous resets can create a first-of-month spike, but this is manageable at small scale.
Avoid arbitrary windows that do not match how customers view usage. If they expect monthly payment while your window resets daily, support will hear that they ran out on day 3.
Handling Near-Budget and Over-Budget Conditions
A hard cutoff that returns 429 is the right default, but two situations can make it unpleasant for developers.
First, a request may slightly exceed the remaining budget. With 200 tokens left and a 500-token system prompt, reject it for accuracy or allow it and flag the overage for a gentler but less precise result. We support both choices at the route. Paid applications that bill token cost back should reject. Internal tools may prefer a small overage allowance to avoid unexpected failures.
Second, a stream can exhaust the budget before its final token count is known. This is a real issue for streaming-first LLM APIs. Our current approach checks an estimated count before sending, based on request payload size, then reconciles actual usage when the stream ends. The estimate is conservative, so some requests near the boundary are rejected even though they might have fit. Letting every stream through and reconciling post-hoc instead produces systematic over-budget conditions.
Surfacing Budget State to Consumers
A consumer receiving 429 without context will likely contact support. Response headers can expose budget state at almost no cost and remove many such inquiries.
X-Token-Budget-Limit: 500000
X-Token-Budget-Remaining: 12430
X-Token-Budget-Reset: 2025-12-01T00:00:00Z
Use the same convention as X-RateLimit-* headers for request limits. Integrations can read them for backoff and quota management. Send them on every response, not only 429 responses, so consumers can see their budget during normal operation.
Gateway token enforcement needs no application-code changes and does not require every upstream model provider to offer matching cost controls. Usage is already in the response body. The gateway reads, aggregates, and enforces it. That separation matters: spend control belongs to infrastructure between the application and model, not to application code.