Tune wait budgets, pacing, and cooldowns

Purpose

Set a wait budget that protects response time without wasting preferred capacity. Configure pacing only where Relay can release requests at useful intervals.

Prerequisites

• A Group contains at least two ranked Models.
• The Models have known or observed limits.
• Fallback is enabled when lower-ranked Models are acceptable.

Set the wait budget

1. Open Groups.
2. Edit the target Group.
3. Set Maximum wait to the longest queue delay that callers accept.
4. Save the Group.

Relay waits for the preferred Model when its predicted eligibility is inside this budget. Relay evaluates lower-ranked Models when the preferred wait exceeds this budget and fallback is enabled.

Configure request pacing

1. Open Limits.
2. Add a request limit with a second or minute period.
3. Enable pacing on the Model.
4. Send a controlled request series.
5. Open Realtime and confirm that requests move at the expected interval.

Pacing applies only to request limits over second and minute periods. Hour, day, and month request limits are hard caps. Token and spend limits are not paced.

Read cooldown state

A cooldown records the next time that a resource can be eligible. Relay can derive a cooldown from:

• Configured or observed request capacity.
• Provider Retry-After information.
• Bounded throttle backoff.
• Provider health after connection, timeout, or 5xx failures.

A user-specific or API-key-specific limit does not cool the shared Provider or Model. It defers or rejects only that principal's request.

Test two outcomes

Wait for the preferred Model

Use a cooldown shorter than the Group maximum wait. Confirm that the request remains queued for the preferred Model.

Select a fallback

Use a predicted preferred wait longer than the Group maximum wait. Confirm that Relay selects the next eligible ranked membership.

Expected result

Relay waits when the preferred Model fits the wait budget. It evaluates a fallback only when the preferred wait exceeds that budget.

Verification

Inspect Queue, Realtime, and Logs. The selected route, predicted eligibility, wait duration, and fallback count must describe the same decision.

Recovery

• If requests reject too early, inspect the effective wait budget and tighter applicable limits.
• If requests wait too long, reduce the Group maximum wait.
• If capacity releases later than expected, inspect conservative long-window buckets and observed limits.

Next

Configure guardrails before and after a model call.