José Agretti
09/23/2025, 10:28 PMMarvin
09/23/2025, 10:28 PMMarvin
09/23/2025, 10:34 PMprefect work-queue create high --pool <your-vertex-pool> --priority <higher>
prefect work-queue create low --pool <your-vertex-pool> --priority <lower>
2) Deploy the same flow twice, each pointing at a different queue
from prefect import flow
@flow
def my_flow():
...
# High priority deployment
my_flow.deploy(
name="my-flow-high",
work_pool_name="<your-vertex-pool>",
work_queue_name="high",
)
# Low priority deployment
my_flow.deploy(
name="my-flow-low",
work_pool_name="<your-vertex-pool>",
work_queue_name="low",
)
3) Start workers for that pool (they can poll both queues)
prefect worker start -p <your-vertex-pool> -q high -q low
Notes:
- Priority is defined on the work queue; workers will pull from higher-priority queues first within the same pool.
- If your high-priority runs also need different infra (e.g., larger machines/GPUs on Vertex AI), use separate deployments with different job variables, or even separate work pools, to change resources alongside priority.
- Tags/parameters do not route a run to a different queue; queue selection is set on the deployment.
Docs:
- Work pools and queues in Prefect 3.x
- Concurrency limits (helpful if you also want to cap throughput per queue/pool)
If you want, share your pool/queue names and I can sketch the exact commands for your Vertex AI setup.