# Handle provider 429 responses

Canonical HTML: https://anchorshell.com/guides/handle-provider-429s/

Test Relay cooldown and requeue behavior without creating a retry storm.

Last updated: 2026-08-18

## Purpose

Verify that Relay controls provider retries after an upstream rate limit. Callers must not add an independent retry loop around this behavior.

## Prerequisites

- Relay is running.
- The built-in dummy provider uses `http://localhost:11730/v1/dummy`.
- The dummy provider credential uses your `RELAY_API_TOKEN` (its endpoint shares inference protection).
- A model named `dummy` routes to the dummy provider.
- You can inspect Realtime, Queue, and Logs.

## Trigger a controlled 429

1. Send a request that tells the dummy provider to return `429`.

```sh
curl -i http://localhost:11730/v1/chat/completions \
  -H 'Authorization: Bearer <RELAY_API_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d '{"model":"dummy","messages":[{"role":"user","content":"ratelimit_429"}]}'
```

**Response — 200 OK**

Example non-streaming response (selected fields). Content, model, and usage depend on the selected provider.

```json
{
  "id": "<COMPLETION_ID>",
  "object": "chat.completion",
  "model": "gpt-5-mini",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Relay waits for the preferred model before considering fallback."
      },
      "finish_reason": "stop"
    }
  ]
}
```

2. Open **Realtime**.

3. Find the selected model. Its state changes to a rate-limited cooldown.

4. Open **Queue**. The same task remains under scheduler control while the model cools.

5. Open **Logs**. Confirm that the provider result and cooldown reason are present without a stored provider secret or authorization value.

## Expected result

A provider `429` does not cause immediate fallback for the affected task. Relay uses this order to calculate the delay:

1. Use a valid `Retry-After` header.
2. Otherwise, derive spacing from a known request limit.
3. Otherwise, use the bounded fallback delay.

Relay marks the selected model as cooling down. Relay then requeues the same task on that model. Later candidate selection can avoid the model while its cooldown is active.

Connection failures, maximum-latency timeouts, and upstream `5xx` responses use a different health and failure-fallback path.

## Prevent a retry storm

- Let Relay own the retry schedule.
- Set realistic configured limits when you know the provider contract.
- Keep observed-limit learning enabled when the provider sends usable rate headers.
- Set each group wait budget to the maximum delay that its callers accept.
- Add ranked alternatives only when fallback is appropriate for that workload.

## Verification

The model stays in cooldown until its next eligible time. The queued task does not create independent provider calls during that interval.

## Recovery

If requests remain delayed after the expected time, inspect the model cooldown, configured limits, observed limits, and group wait budget. A tighter limit can extend eligibility beyond the provider header.

## Next

[Inspect a request trace and reconcile cost](https://anchorshell.com/guides/inspect-request-trace).