Docs

LLM Routing

The llm-routing-inbound policy routes requests across prioritized providers. It automatically fails over on errors or timeouts.

Provider setup

Configure a providers array in the policy options. Each entry defines an upstream URL, auth header injection, and routing weight. Providers run in priority order.

{
  "name": "llm-router",
  "policyType": "llm-routing-inbound",
  "options": {
    "providers": [
      {
        "name": "openai-primary",
        "upstream": "https://api.openai.com/v1/chat/completions",
        "authHeader": "Bearer $env(OPENAI_API_KEY)",
        "weight": 1.0,
        "priority": 1
      },
      {
        "name": "anthropic-fallback",
        "upstream": "https://api.anthropic.com/v1/messages",
        "authHeader": "x-api-key $env(ANTHROPIC_API_KEY)",
        "weight": 0.0,
        "priority": 2  // only used on failover
      }
    ]
  }
}

Failover rules

A provider fails on a 429, 500, 502, 503, or 504 response, or a connection timeout. Zuplo retries within the same request context against the next provider, so the caller never sees the retry.

"retryConfig": {
  "maxAttempts": 3,          // total attempts including first try
  "timeoutMs": 8000,         // per-provider timeout
  "failoverOnStatusCodes": [429, 500, 502, 503]
}

If every provider is exhausted, the gateway returns 502 Bad Gateway with an X-Zuplo-Routing-Attempts header listing the providers tried.

Weighted routing

Assign weight values to multiple providers to distribute traffic by normalized probability. Use weights for A/B testing or traffic shifts.

"providers": [
  {
    "name": "gpt-4o",
    "upstream": "https://api.openai.com/v1/chat/completions",
    "authHeader": "Bearer $env(OPENAI_API_KEY)",
    "weight": 0.7  // 70% of traffic
  },
  {
    "name": "claude-3-5",
    "upstream": "https://api.anthropic.com/v1/messages",
    "authHeader": "x-api-key $env(ANTHROPIC_API_KEY)",
    "weight": 0.3  // 30% of traffic
  }
]

Latency-based routing

Set "strategy": "latency" to prefer the provider with the lowest P50 latency from the last 60 seconds. Without recent traffic, routing falls back to round-robin.

{
  "name": "llm-router",
  "policyType": "llm-routing-inbound",
  "options": {
    "strategy": "latency",  // "priority" | "weighted" | "latency"
    "providers": [ ... ]
  }
}

Request formats

For OpenAI and Anthropic API differences, set "normalize": true. Zuplo translates OpenAI chat completions to Anthropic messages and normalizes the response back to OpenAI format.

"normalize": true,   // translate request/response to OpenAI format
"modelMap": {
  "gpt-4o": "claude-3-5-sonnet-20241022"  // map model names per-provider
}

Routing headers

Header	Value
`X-Zuplo-Provider`	Name of the provider that served the request
`X-Zuplo-Routing-Attempts`	Comma-separated list of providers tried (included when failover occurred)
`X-Zuplo-Routing-Strategy`	Routing strategy used: `priority`, `weighted`, or `latency`