Build a free, local, and paid model order
Create an explicit model order that uses free or local capacity before a paid fallback.
Purpose
Create a Standard Group that prefers selected free or local capacity and uses a paid model only when the wait budget permits fallback.
Relay follows the order that you configure. Relay does not discover the cheapest model or optimize this order automatically.
Prerequisites
Add and verify these model types:
- A free or subscription-backed model with known capacity.
- A local OpenAI-compatible model.
- A paid API model.
Add configured pricing if you want Relay to estimate cost or enforce spend limits.
Create the order
- Open Groups.
- Add a Standard Group named
free-local-paid. - Enable fallback.
- Set the maximum wait that the caller accepts.
- Add the free model at rank
1. - Add the local model at rank
2. - Add the paid model at rank
3. - Save the group.
Add capacity controls
- Add the known request or token limit to the free model.
- Enable request pacing for second- or minute-level request limits when smooth spacing is useful.
- Add a concurrency limit to the local model if local hardware has a fixed parallel capacity.
- Add a maximum-latency value when a hanging upstream must be cancelled.
Understand selection
| Condition | Relay action |
|---|---|
| Rank 1 is eligible now | Select rank 1 |
| Rank 1 becomes eligible inside the wait budget | Wait for rank 1 |
| Rank 1 exceeds the budget and rank 2 fits | Select rank 2 |
| Ranks 1 and 2 exceed the budget and rank 3 fits | Select rank 3 |
| No candidate fits | Return 429 with Retry-After |
An upstream 429 for an active task cools the selected model and requeues that same task. It does not automatically move that task to the next rank.
Expected result
Relay prefers the configured free or local Models. It selects the paid Model only when the higher-ranked Models cannot serve inside the wait budget.
Verification
Use Realtime to confirm each model's limit, cooldown, and selected state. Use Logs to confirm the group, candidate evaluation, selected route, wait time, and cost.
Recovery
If the paid model receives traffic too early, increase the group wait budget or inspect the higher-ranked models' limits and cooldowns.