Zach Chin
10/03/2025, 1:53 AMwork_queue_name to a work_pool but the UI and CLI allows for modifying a work_queue to set it's priority. How can I do this with prefect.yaml?
For example:
deployments:
- name: deployment-a
entrypoint: src/flows.py:flow_a
work_pool:
name: my_work_pool
work_queue_name: default
job_variables: {}
- name: deployment-b
entrypoint: src/flows.py:flow_b
work_pool:
name: my_work_pool
work_queue_name: urgent
job_variables: {}
the default work_queue seems to get a priority of 1 by default, how to i configure that to 5 and create an urgent work_queue with priority of 1?Marvin
10/03/2025, 1:53 AMMarvin
10/03/2025, 1:58 AMprefect.yaml only accepts work_pool.name, work_queue_name, and job_variables for a deployment. You cannot set work queue properties (like priority or concurrency limit) in prefect.yaml.
What to do instead:
- Create/configure work queues via the UI or CLI, or programmatically with the Python client/API.
- Then reference those queues from your deployments in prefect.yaml with work_queue_name.
Notes on priority:
- Lower numbers are higher priority (1 is highest).
- New queues default to priority 1 unless you set a different value when creating/updating them.
CLI examples (create queues with desired priorities)
# Create the queues in your pool with the priorities you want
prefect work-queue create default --pool my_work_pool --priority 5
prefect work-queue create urgent --pool my_work_pool --priority 1
# (optional) set a concurrency limit
prefect work-queue set-concurrency-limit default 10 --pool my_work_pool
We verified these commands with --help. See prefect work-queue create --help and prefect work-queue set-concurrency-limit --help.
Updating an existing queue’s priority
- The CLI does not currently expose a “set priority” command for existing queues.
- Use the UI or a tiny Python script with the client:
import asyncio
from prefect.client.orchestration import get_client
async def ensure_queue_priority(pool, name, priority):
async with get_client() as client:
try:
q = await client.read_work_queue_by_name(name, work_pool_name=pool)
await client.update_work_queue(id=q.id, priority=priority)
except Exception:
await client.create_work_queue(name=name, work_pool_name=pool, priority=priority)
async def main():
await ensure_queue_priority("my_work_pool", "default", 5)
await ensure_queue_priority("my_work_pool", "urgent", 1)
asyncio.run(main())
Suggested workflow
- As a pre-deploy step (in CI/CD), run the CLI commands (or the script) once to create/update your pool’s queues with the right priorities.
- Keep your prefect.yaml deployments pointing at those queues:
deployments:
- name: deployment-a
entrypoint: src/flows.py:flow_a
work_pool:
name: my_work_pool
work_queue_name: default
job_variables: {}
- name: deployment-b
entrypoint: src/flows.py:flow_b
work_pool:
name: my_work_pool
work_queue_name: urgent
job_variables: {}
Docs for reference:
- Work pools and queues: Work Pools
- Creating deployments: Create deployments
If you want, I can share a small shell script you can drop into CI that idempotently creates the queues with the right priority before prefect deploy.