Inference State
Live queue pressure and recorded prompt-cache evidence for All models.
Queue stats are a live process observation of one observed replica; they are not fleet-wide totals.
KV warmth is recorded historical evidence from request logs; it is not a claim about current physical KV residency.
Live response from one load-balanced queue process
Pending entries across all subscribed queue replicas. Hold pauses launch without losing arrival position; move-to-front stays inside the entry's tier; cancel is terminal.
TTL'd operator overrides that prefer or exclude one provider for a cache or affinity identity. Expiry returns the identity to free rendezvous.
In-flight generations on this gateway process. Kill frees the provider slot; the warm KV prefix is kept and the caller's durable queue retries where one exists.