Queue and pacing

Queue states

Relay uses heap-backed ready and delayed queues.

• Queued / waiting work has been accepted but is not yet eligible.
• Ready work can be considered for dispatch.
• In flight work has an active upstream request.
• Completed, failed, and cancelled are terminal request-log states.

The queue is in memory and does not survive a process restart. Configuration and request logs are stored separately.

Eligibility

The resolver calculates the next eligible time across configured and observed policies for global, provider, endpoint, and lane scopes. Hosted Relay merges user-scoped policy timing into the same decision.

Request, token, spend, and concurrency needs can affect eligibility. Estimated tokens and cost may be reserved while work waits, based on the current settings, then reconciled when terminal usage is known.

Pacing

Short request windows can be paced rather than released as one burst. When a request becomes ready, Relay reevaluates eligibility before dispatch. Work that is still blocked returns to the delayed queue instead of stampeding an upstream.

The scheduler can skip user-deferred work and continue considering other eligible requests. User-specific limits do not place a provider or endpoint into a global cooldown.