OpenAI adjusted gpt-4o pricing in Q2 2025, and Anthropic cut claude-3-5-sonnet pricing soon after. A code-based routing choice requires a deployment when prices or latency change. A config-based choice lets you merge a PR instead. Provider latency also varies by region and time of day, often without notice.
Config-as-code routing follows a broader infrastructure rule: frequent decisions belong outside the application layer, where teams can review, version, and roll them back separately from business logic.
What Routing Config Contains
A gateway routing rule defines three things: the request shape that matches, the upstream provider to use, and the response when that provider is unavailable or errors. That is the complete surface.
{
"route": "POST /api/complete",
"upstream": {
"provider": "openai",
"model": "gpt-4o-mini",
"apiKeySecret": "env:OPENAI_API_KEY",
"fallback": [
{
"provider": "anthropic",
"model": "claude-haiku-3-5",
"apiKeySecret": "env:ANTHROPIC_API_KEY",
"onStatus": [429, 500, 502, 503]
}
]
}
}The onStatus field sets the fallback condition. When the primary provider returns 429, the gateway retries against the fallback without exposing a failure to the consumer. The same happens for 500 or 503. The gateway's routing event log shows which provider handled each request.
Cost-Based Routing
Request shapes carry different costs. A summarization task needing strong reasoning costs more on a frontier model than a brief classification or extraction task. Cost routing matches each task's complexity with the least expensive model tier that meets its quality bar.
One option is route-level model selection. Create separate gateway routes for each task type, with each route targeting the suitable model tier:
[
{
"route": "POST /api/reason",
"upstream": {
"provider": "openai",
"model": "gpt-4o",
"apiKeySecret": "env:OPENAI_API_KEY"
}
},
{
"route": "POST /api/classify",
"upstream": {
"provider": "openai",
"model": "gpt-4o-mini",
"apiKeySecret": "env:OPENAI_API_KEY",
"fallback": [
{
"provider": "anthropic",
"model": "claude-haiku-3-5",
"apiKeySecret": "env:ANTHROPIC_API_KEY",
"onStatus": [429, 500, 503]
}
]
}
}
]This fits an API whose routes identify task types. The client selects the endpoint. If the gateway must inspect request content instead, custom policy logic has to read the request body, making the setup more complex.
Latency-Based Routing and Failover
Provider latency changes over time. OpenAI's P95 response time for gpt-4o during peak US business hours exceeds its off-peak time. Anthropic follows different geographic patterns. For latency-sensitive applications, one static primary may be the wrong choice at some times of day.
A practical fallback chain handles this without active latency monitoring. When a primary is slow enough to time out, the resulting 500 or 504 response triggers fallback. This reacts to request errors rather than actively checking health or adjusting weights. For most applications, that is sufficient.
Active latency routing, sending traffic to the fastest provider by recent response times, needs a load balancing policy that tracks rolling latency per provider and changes weights. This fits cases where an SLA demands sub-second p95 latency and the budget supports routing to multiple providers simultaneously.
Load Balancing Across Multiple Keys
With multiple API keys for one provider, perhaps across environments or after buying capacity, you can spread requests across keys and remain within per-key rate limits:
{
"upstream": {
"provider": "openai",
"model": "gpt-4o",
"loadBalance": {
"strategy": "round-robin",
"backends": [
{ "apiKeySecret": "env:OPENAI_KEY_1", "weight": 50 },
{ "apiKeySecret": "env:OPENAI_KEY_2", "weight": 50 }
]
}
}
}Round-robin shares requests evenly. Weighted distribution sends more traffic through a key with higher limits and uses a secondary key for overflow. The gateway rotates keys without application code changes.
What Fallback Looks Like in Practice
After a primary provider returns 429, the gateway marks the request for fallback and retries it with the next provider. The consumer waits slightly longer, adding the failed attempt's latency to the fallback latency, but receives a successful response. The client's response is 200.
The routing event log records the selected provider, whether fallback ran, the provider that ultimately returned the response, and latency for each leg. It can reveal a primary returning enough 429s to make the fallback the real primary. A fallback rate above 5% over a rolling hour suggests negotiating higher limits with the primary provider or reconsidering the primary choice.
What This Approach Does Not Handle
Provider routing fits request types where providers are semantically interchangeable. It breaks down when a prompt or system message depends on one model's behavior. A prompt tuned for GPT-4o's instruction-following style may behave noticeably differently on Claude, and that difference may or may not matter to your application.
Test critical prompts against the fallback model before enabling fallback routing in production. Acceptable output gives you added resilience without a quality cost. Otherwise, tune the prompt for multiple providers, or accept different fallback output characteristics and handle them in the application.
The routing config manages traffic distribution. Provider equivalence for your use case is a content question outside the gateway.