Five Rate Limit Patterns and When to Use Each

Naive rate limiting breaks when real traffic arrives. Teams often choose a pattern, ship it, then meet its failure mode when a bursty workload or forceful integration reaches the service. This guide covers what the five main patterns do at the counter level, where each works, and what can go wrong.

Pattern 1: Fixed Window Counter

The simplest model: divide time into windows of fixed size (1 minute, 1 hour, 1 day), increment a counter on each request, reject when the counter exceeds the limit, reset the counter at the window boundary.

Window:  |--- 60s ---|--- 60s ---|--- 60s ---|
Count:       45/100     100/100    0/100 → burst

Its weakness is the boundary burst. A consumer can send 100 requests in the final second of window N, then 100 in the first second of window N+1, allowing 200 requests within 2 seconds. The atomic reset ignores traffic density.

Fixed windows suit daily quotas. If the rule says "this API key may make 10,000 calls per calendar day", and the contract defines a quota rather than a throughput ceiling, fixed window fits. A daily quota measures volume across the day, so its boundary burst is acceptable.

It is a poor choice when momentary throughput matters: real-time APIs, LLM proxies with per-call costs, or any service where 200 requests in 2 seconds can harm downstream systems.

Pattern 2: Sliding Window Log

Store a timestamped entry for every request. For each new request, count entries inside the window ending now. Reject the request when that count exceeds the limit, then remove old entries.

This is the most precise pattern. It has no boundary burst: a consumer stays within N requests during every rolling N-minute window.

The trade-off is memory and compute per request. The gateway stores and queries a log for each API key. Short windows and high request volumes can make that costly. For most LLM API workloads, whose rates are orders of magnitude below a typical web API, the cost is negligible. At 100,000 requests per second, it is not.

We recommend sliding window log for per-key token budgets on LLM routes. Token cost is the important measure, and precise accounting justifies the overhead at typical LLM call volumes.

Pattern 3: Sliding Window Counter

This hybrid keeps two neighboring fixed-window counters and estimates a sliding window by weighting them. If the current window is 40% into a 60-second period, multiply the previous counter by 60%, leave the current at 100%, and add them.

Prev window count: 80  (weight: 40% * 80 = 32)
Curr window count: 30  (weight: 100% * 30 = 30)
Effective count:   62  ← compare against limit

The estimate has bounded error, usually within a few percent, while memory stays constant regardless of request rate. Redis uses this pattern in its rate limiter implementation.

Use it for high-throughput APIs that need sliding-window behavior without log-per-key memory costs. Its error suits most production cases. Choose sliding window log for exact, billing-grade accounting, and sliding window counter for smooth limiting at scale.

Pattern 4: Token Bucket

Each key receives a bucket with a maximum capacity, its burst capacity. Tokens return at a constant rate, measured in tokens per second. Each request spends one or more tokens. With too few tokens, reject or queue it according to policy.

Bucket capacity: 20 tokens
Refill rate:     5 tokens/sec
Burst of 20 requests at t=0:  all consume from bucket (bucket empties)
Request at t=1:  4 tokens refilled, passes
Steady state:    consumer can sustain 5 req/sec indefinitely

Token bucket deliberately permits bursts up to its capacity. That often matches developer needs: send a batch at once, then continue at a steady rate. An idle key builds capacity, which makes the pattern friendly to developers.

The trade-off is that the window concept does not set the maximum burst. A key idle for an hour can build up to the bucket capacity, not one hour's tokens. Set the burst ceiling with capacity, rather than indirectly with window size.

For LLM API routes, a request may use 4,000 tokens while the gateway meters output tokens. Token bucket with variable cost per request fits naturally. Set each request's cost to the LLM token count it uses.

Pattern 5: Concurrent Connection Limit

Count in-flight requests for each key. Reject new ones when that count exceeds the limit, then return the slot when the response finishes.

This pattern addresses a different dimension. It limits parallel work, not throughput over time. A slow request taking 30 seconds keeps its slot for the full duration.

For LLM proxies, concurrent limits can matter more than throughput limits. An LLM call may last several seconds or several minutes. A consumer sending 50 simultaneous LLM requests creates different load from one sending 50 requests per second in serial. The first holds 50 connections and 50 upstream slots waiting. A throughput-only limit misses that difference.

Pair a concurrent limit with a windowed pattern. Concurrency protects against running out of connections, while the windowed limit addresses sustained high-rate abuse.

Composing Patterns at the Gateway

In practice, combine several patterns. A Zuplo route for an LLM chat endpoint might look like this:

{
  "policies": [
    {
      "name": "concurrent-limit",
      "handler": "concurrent-limit-inbound",
      "options": { "key": "api-key", "max_concurrent": 5 }
    },
    {
      "name": "token-bucket-per-key",
      "handler": "token-bucket-inbound",
      "options": {
        "key": "api-key",
        "capacity": 20,
        "refill_rate": 2,
        "refill_unit": "second"
      }
    },
    {
      "name": "daily-quota",
      "handler": "fixed-window-inbound",
      "options": {
        "key": "api-key",
        "window": "1d",
        "max": 1000
      }
    }
  ]
}

The concurrent check runs first to block simultaneous connection floods. Token bucket handles bursts and steady throughput. The daily quota upholds the billing contract.

Policy order matters when checks are stacked. Run cheaper checks first. A concurrent check is a counter lookup without time-window math. Place it ahead of complex policies so clearly over-limit requests stop before extra work runs.

Choose a pattern by matching protection to the workload's risk. Ask what harmful traffic looks like on this route: sustained hammering, a sudden burst, slow parallel connections, or daily quota exhaustion? That answer points to the pattern, or combination, to put in the config.