<@ULVA73B9P> Can I adjust the priority of flow run...
# ask-marvin
u
@Marvin Can I adjust the priority of flow runs in a Prefect work queue? Additionally, are 'late' states typically caused by hitting concurrency limits?
m
Error: unhandled errors in a TaskGroup (1 sub-exception)
Sorry, I encountered an error while processing your request. Please try again.
u
@Marvin Can I adjust the priority of flow runs in a Prefect work queue? Additionally, are 'late' states typically caused by hitting concurrency limits?
m
thought for 180.9 seconds
Short answer - Priority: Yes, but it’s set on work queues (not individual flow runs or deployments). Lower integers mean higher priority. Workers poll higher‑priority queues first. - Late states: Not directly “caused by” concurrency limits, but concurrency limits can keep runs from moving to PENDING, which can lead to Late if the grace period elapses. Details and how-tos (Prefect 3.x) - How priority works - Priority is configured per work queue. There is no per‑run or per‑deployment priority knob. To “prioritize” a deployment’s runs, point it at a higher‑priority queue. - Workers select work across queues in order of queue priority, then by the run’s schedule. - Set/adjust queue priority - CLI (create a queue with a priority):
Copy code
prefect work-queue create critical --pool my-pool --priority 1
- Python API (create/update):
Copy code
from prefect.client.orchestration import get_client

    async with get_client() as client:
        # create
        await client.create_work_queue(
            name="critical",
            work_pool_name="my-pool",
            priority=1
        )

        # update
        await client.update_work_queue(
            id=<queue_id>,
            priority=3
        )
- What makes a run Late? - A run is marked Late when its scheduled start time has passed and it hasn’t transitioned to PENDING within a small window (15s by default). - Common causes: - No healthy/active workers on the target work pool - Worker polling the wrong pool/queue, or the queue is paused - Workers at capacity (e.g., pool/queue/deployment concurrency limits fully used) - Concurrency relationship: - Hitting a concurrency limit moves the run into an “awaiting concurrency slot” scheduled state; if it waits past the Late threshold, it becomes Late. So concurrency contributes indirectly. - Tuning and checks - Adjust Late threshold:
Copy code
export PREFECT_API_SERVICES_LATE_RUNS_AFTER_SECONDS=30
- Set/clear queue concurrency limits (if you’re seeing backlog/Late due to capacity):
Copy code
prefect work-queue set-concurrency-limit 5 critical --pool my-pool
    prefect work-queue clear-concurrency-limit critical --pool my-pool
Docs - States (Late definition and lifecycle): States - Work pools/queues and priority: Work pools and queues - Managing work pools/queues (CLI): Manage work pools and queues - Concurrency limits overview: Global concurrency limits If you share how your deployments and queues are set up (which pool/queue, queue priorities, any concurrency limits, and worker status), I can help spot why you’re seeing Late runs and suggest concrete tweaks.