The deployment script had changed, and nobody knew it. In 2013, configuring an API gateway meant editing nginx.conf on a VM, running a syntax check, and hoping the jump box script still worked as it had for the last person. Backends were IP addresses kept in a spreadsheet. Adding or losing one meant editing the config and reloading nginx. The gateway was a long-lived stateful artifact that knew too much about the topology in front of it.
That design carried a coherent set of assumptions. They made sense then. Almost none hold up for LLM workloads.
The Static Topology Assumption
Pre-container API gateways assumed backend services were known, stable, and addressable by IP. You put them in the config. Changes were rare and treated as infrastructure work, rather than routine operations.
Nginx upstream blocks, F5 pools, and HAProxy backend configurations all assumed you could list every server receiving traffic. The gateway was a routing table with policies attached. Know the routes, configure the gateway. Its topology was static by design.
Apigee and similar API management platforms added a service layer: a UI for defining API products, applying policies, and handling developer access. Routing still depended on backend URLs someone entered and stored in a database. The topology changed more slowly, not more quickly.
The Uniform Request Cost Model
Pre-container rate limiting was usually measured in requests per second or requests per minute. The assumption was simple: each request used about the same backend resources, took about the same time, and cost about the same to serve.
That worked for REST APIs serving database-backed resources. GET /api/orders and GET /api/users returned different data, but each involved a database query and JSON serialization, took 5 to 50ms, and was inexpensive. Requests per second reasonably represented load.
For that model, fixed window counters and token buckets measured in requests were sufficient. Capacity could be expressed as "this service can handle 1,000 requests per second", then limits could follow.
What Kubernetes Changed
Kubernetes introduced dynamic service discovery for backend topology. Pods appeared and disappeared. Backends were ephemeral. An IP serving traffic at noon could be gone by 1pm. Gateways had to stop tracking individual IPs and discover backends through the service registry.
Ingress controllers, including nginx-ingress, Traefik, and later Kong and Ambassador, became the usual gateway layer for Kubernetes workloads. Config moved into YAML manifests and joined the GitOps workflow: you PR'd an ingress change, CI applied it, and routing updated without manual work.
That was a major improvement. Topology became dynamic, config was version-controlled, and deployment was automated. Yet assumptions about request cost and backend ownership stayed put. Kubernetes addressed topology, not the rate limiting or ownership model.
What LLM Workloads Break
LLM workloads break four assumptions that survived the Kubernetes transition.
Non-uniform request cost. An LLM call with 8,000 input tokens and a 1,000-token completion costs roughly 40 to 50 times more than a call with 200 input tokens and a 100-token completion. Both arrive as POST /api/chat. Both look like one request. Measuring in requests per minute is not a useful proxy for load or cost. You need to measure in tokens.
The backend is not yours. When you proxy to OpenAI or Anthropic, the backend is a third-party API with its own rate limits, pricing structure, and service agreement. The gateway is now mediating access to an external resource you do not own and cannot scale. You cannot add capacity by spinning up more pods. You negotiate rate limit tiers with the provider or distribute load across multiple keys.
Behavior without deployment. OpenAI can ship model updates on the same endpoint without adding an endpoint path. gpt-4o-2025-01-15 and gpt-4o-2025-04-14 are different models and may behave differently, yet both use /v1/chat/completions with "model": "gpt-4o". The old idea, "if the endpoint is the same, behavior is the same," fails here. Routing config can stay identical while the model's output changes.
Long-lived connections. Streaming completions and MCP sessions create persistent connections measured in seconds to minutes. An nginx ingress controller configured with a 60-second proxy timeout will silently kill a streaming response that takes 90 seconds to complete. The default timeouts tuned for sub-second REST calls are wrong for LLM workloads, and they fail silently: the client sees a dropped connection, not a timeout error.
The Assumption That Needs Replacing
Retire the idea that a request is uniform work you produce and serve. For LLM proxies, it is access to a metered external resource, measured in consumed tokens rather than established connections. Gateways need token budgets instead of request counters, access management for third-party capacity instead of only routing to owned services, and tracking for backend behavior that changes without deployment.
Pre-container API gateway designs were not wrong for their time. They matched the job: manage access to your services, expose consistent schemas to known consumers, and apply policies at the HTTP boundary. Their assumptions fit the constraints that mattered then.
LLM workloads bring different constraints. A suitable gateway measures tokens, routes among third-party providers, treats streaming as normal, and watches for model behavior changes. That calls for different config, not a louder old one.