<@ULVA73B9P> Can I define a work queue configurati...
# ask-marvin
z
@Marvin Can I define a work queue configuration in prefect.yaml? The documentation shows how to assign a
work_queue_name
to a
work_pool
but the UI and CLI allows for modifying a work_queue to set it's priority. How can I do this with prefect.yaml? For example:
Copy code
deployments:
  - name: deployment-a
    entrypoint: src/flows.py:flow_a
    work_pool:
      name: my_work_pool
      work_queue_name: default
      job_variables: {}
  - name: deployment-b
    entrypoint: src/flows.py:flow_b
    work_pool:
      name: my_work_pool
      work_queue_name: urgent
      job_variables: {}
the
default
work_queue seems to get a priority of 1 by default, how to i configure that to 5 and create an
urgent
work_queue with priority of 1?
m
thought for 321.5 seconds
Short answer: not today. In Prefect 3.x,
prefect.yaml
only accepts
work_pool.name
,
work_queue_name
, and
job_variables
for a deployment. You cannot set work queue properties (like priority or concurrency limit) in
prefect.yaml
. What to do instead: - Create/configure work queues via the UI or CLI, or programmatically with the Python client/API. - Then reference those queues from your deployments in
prefect.yaml
with
work_queue_name
. Notes on priority: - Lower numbers are higher priority (1 is highest). - New queues default to priority 1 unless you set a different value when creating/updating them. CLI examples (create queues with desired priorities)
Copy code
# Create the queues in your pool with the priorities you want
prefect work-queue create default --pool my_work_pool --priority 5
prefect work-queue create urgent  --pool my_work_pool --priority 1

# (optional) set a concurrency limit
prefect work-queue set-concurrency-limit default 10 --pool my_work_pool
We verified these commands with
--help
. See
prefect work-queue create --help
and
prefect work-queue set-concurrency-limit --help
. Updating an existing queue’s priority - The CLI does not currently expose a “set priority” command for existing queues. - Use the UI or a tiny Python script with the client:
Copy code
import asyncio
from prefect.client.orchestration import get_client

async def ensure_queue_priority(pool, name, priority):
    async with get_client() as client:
        try:
            q = await client.read_work_queue_by_name(name, work_pool_name=pool)
            await client.update_work_queue(id=q.id, priority=priority)
        except Exception:
            await client.create_work_queue(name=name, work_pool_name=pool, priority=priority)

async def main():
    await ensure_queue_priority("my_work_pool", "default", 5)
    await ensure_queue_priority("my_work_pool", "urgent", 1)

asyncio.run(main())
Suggested workflow - As a pre-deploy step (in CI/CD), run the CLI commands (or the script) once to create/update your pool’s queues with the right priorities. - Keep your
prefect.yaml
deployments pointing at those queues:
Copy code
deployments:
  - name: deployment-a
    entrypoint: src/flows.py:flow_a
    work_pool:
      name: my_work_pool
      work_queue_name: default
      job_variables: {}
  - name: deployment-b
    entrypoint: src/flows.py:flow_b
    work_pool:
      name: my_work_pool
      work_queue_name: urgent
      job_variables: {}
Docs for reference: - Work pools and queues: Work Pools - Creating deployments: Create deployments If you want, I can share a small shell script you can drop into CI that idempotently creates the queues with the right priority before
prefect deploy
.