Rate Limiting

Zuplo supports request-count limits and LLM token budgets. Both run as inbound policies before requests reach the upstream provider.

Request rate limiting

Use rate-limit-inbound to limit a consumer's requests within a time window. It can key limits by consumer identity, IP, or a custom selector.

{
  "name": "rate-limit",
  "policyType": "rate-limit-inbound",
  "options": {
    "rateLimitBy": "consumer",       // "consumer" | "ip" | "global"
    "requestsPerWindow": 60,
    "windowSizeInSeconds": 60,
    "algorithm": "sliding-window"   // "fixed-window" | "sliding-window" | "token-bucket"
  }
}

Algorithm options

• sliding-window (default): most accurate; prevents burst through window boundaries
• fixed-window: lowest overhead; resets at the clock boundary and may burst then
• token-bucket: allows burst up to burstCapacity while keeping average rate at limit

"algorithm": "token-bucket",
"requestsPerWindow": 60,
"windowSizeInSeconds": 60,
"burstCapacity": 10   // allow up to 10 extra requests in a burst

LLM token budgets

For LLM proxies, request count poorly reflects cost. Use token-budget-inbound for a daily token limit per consumer. The gateway reads upstream token usage and keeps a rolling daily total.

{
  "name": "daily-token-budget",
  "policyType": "token-budget-inbound",
  "options": {
    "tokenBudgetPerDay": 500000,
    "tokenBudgetBy": "consumer",
    "tokenCountField": "usage.total_tokens",  // JSONPath in upstream response
    "resetAtUtcHour": 0                            // reset at midnight UTC
  }
}

When a consumer's daily token total would exceed its budget, the request gets 429 before forwarding. All responses include X-Zuplo-Token-Budget-Remaining so clients can self-regulate.

Rate limit response headers