Zach Chin
10/03/2025, 1:53 AMwork_queue_name to a work_pool but the UI and CLI allows for modifying a work_queue to set it's priority. How can I do this with prefect.yaml?
For example:
deployments:
- name: deployment-a
entrypoint: src/flows.py:flow_a
work_pool:
name: my_work_pool
work_queue_name: default
job_variables: {}
- name: deployment-b
entrypoint: src/flows.py:flow_b
work_pool:
name: my_work_pool
work_queue_name: urgent
job_variables: {}
the default work_queue seems to get a priority of 1 by default, how to i configure that to 5 and create an urgent work_queue with priority of 1?Marvin
10/03/2025, 1:53 AMMarvin
10/03/2025, 1:58 AMprefect.yaml only accepts work_pool.name, work_queue_name, and job_variables for a deployment. You cannot set work queue properties (like priority or concurrency limit) in prefect.yaml.
What to do instead:
- Create/configure work queues via the UI or CLI, or programmatically with the Python client/API.
- Then reference those queues from your deployments in prefect.yaml with work_queue_name.
Notes on priority:
- Lower numbers are higher priority (1 is highest).
- New queues default to priority 1 unless you set a different value when creating/updating them.
CLI examples (create queues with desired priorities)
# Create the queues in your pool with the priorities you want
prefect work-queue create default --pool my_work_pool --priority 5
prefect work-queue create urgent --pool my_work_pool --priority 1
# (optional) set a concurrency limit
prefect work-queue set-concurrency-limit default 10 --pool my_work_pool
We verified these commands with --help. See prefect work-queue create --help and prefect work-queue set-concurrency-limit --help.
Updating an existing queue’s priority
- The CLI does not currently expose a “set priority” command for existing queues.
- Use the UI or a tiny Python script with the client:
import asyncio
from prefect.client.orchestration import get_client
async def ensure_queue_priority(pool, name, priority):
async with get_client() as client:
try:
q = await client.read_work_queue_by_name(name, work_pool_name=pool)
await client.update_work_queue(id=q.id, priority=priority)
except Exception:
await client.create_work_queue(name=name, work_pool_name=pool, priority=priority)
async def main():
await ensure_queue_priority("my_work_pool", "default", 5)
await ensure_queue_priority("my_work_pool", "urgent", 1)
asyncio.run(main())
Suggested workflow
- As a pre-deploy step (in CI/CD), run the CLI commands (or the script) once to create/update your pool’s queues with the right priorities.
- Keep your prefect.yaml deployments pointing at those queues:
deployments:
- name: deployment-a
entrypoint: src/flows.py:flow_a
work_pool:
name: my_work_pool
work_queue_name: default
job_variables: {}
- name: deployment-b
entrypoint: src/flows.py:flow_b
work_pool:
name: my_work_pool
work_queue_name: urgent
job_variables: {}
Docs for reference:
- Work pools and queues: Work Pools
- Creating deployments: Create deployments
If you want, I can share a small shell script you can drop into CI that idempotently creates the queues with the right priority before prefect deploy.Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by