Routing lanes and fallbacks

Relay Lanes

A Relay Lane groups compatible endpoint/model paths behind one incoming lane name. Memberships have an enabled state and manual rank. Manual ordering is authoritative; Relay does not claim automatic load balancing or AI-driven path scoring.

For each candidate, the scheduler considers:

• endpoint and route compatibility;
• configured and observed limits;
• endpoint cooldown state;
• hosted user, user-model, and user-provider policies;
• estimated token and cost needs;
• an optional maximum estimated request cost;
• the request or lane wait budget.

Relay selects the first candidate that satisfies those conditions. When a candidate exceeds the wait budget and fallback is enabled, selection can continue to another configured path.

Failure fallback

With fallback enabled, connection errors, maximum-latency timeouts, and upstream 5xx failures can mark the selected endpoint cooling or unhealthy and submit the request against remaining candidates.

429 behavior

An upstream 429 or compatible rate-limit response follows a different path. Relay honors Retry-After when available, records the endpoint cooldown, and requeues the affected task. Do not assume that every in-flight 429 immediately moves to another provider. Later candidate selection can avoid an endpoint while it remains in cooldown.

Local-first example

To prefer local capacity, put a compatible local endpoint first in a lane and place cloud paths after it. This is explicit ranked routing—not automatic cost optimization—and it only works for correctly configured, eligible paths.