Handle provider 429 responses

Purpose

Verify that Relay controls provider retries after an upstream rate limit. Callers must not add an independent retry loop around this behavior.

Prerequisites

• Relay is running.
• The built-in dummy provider uses http://localhost:11730/v1/dummy.
• The dummy provider credential uses your RELAY_API_TOKEN (its endpoint shares inference protection).
• A model named dummy routes to the dummy provider.
• You can inspect Realtime, Queue, and Logs.

Trigger a controlled 429

1. Send a request that tells the dummy provider to return 429.

curl -i http://localhost:11730/v1/chat/completions \
  -H 'Authorization: Bearer <RELAY_API_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d '{"model":"dummy","messages":[{"role":"user","content":"ratelimit_429"}]}'

Response — 200 OK

Example non-streaming response (selected fields). Content, model, and usage depend on the selected provider.

{
  "id": "<COMPLETION_ID>",
  "object": "chat.completion",
  "model": "gpt-5-mini",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Relay waits for the preferred model before considering fallback."
      },
      "finish_reason": "stop"
    }
  ]
}

2. Open Realtime.
3. Find the selected model. Its state changes to a rate-limited cooldown.
4. Open Queue. The same task remains under scheduler control while the model cools.
5. Open Logs. Confirm that the provider result and cooldown reason are present without a stored provider secret or authorization value.

Expected result

A provider 429 does not cause immediate fallback for the affected task. Relay uses this order to calculate the delay:

1. Use a valid Retry-After header.
2. Otherwise, derive spacing from a known request limit.
3. Otherwise, use the bounded fallback delay.

Relay marks the selected model as cooling down. Relay then requeues the same task on that model. Later candidate selection can avoid the model while its cooldown is active.

Connection failures, maximum-latency timeouts, and upstream 5xx responses use a different health and failure-fallback path.

Prevent a retry storm

• Let Relay own the retry schedule.
• Set realistic configured limits when you know the provider contract.
• Keep observed-limit learning enabled when the provider sends usable rate headers.
• Set each group wait budget to the maximum delay that its callers accept.
• Add ranked alternatives only when fallback is appropriate for that workload.

Verification

The model stays in cooldown until its next eligible time. The queued task does not create independent provider calls during that interval.

Recovery

If requests remain delayed after the expected time, inspect the model cooldown, configured limits, observed limits, and group wait budget. A tighter limit can extend eligibility beyond the provider header.

Next

Inspect a request trace and reconcile cost.