Use the flag multiple times. Don’t use a comma-separated list. In Prefect 3.x,

--work-queue

(

-q

) is repeatable. Examples:

prefect worker start -p my-pool -q queue-a -q queue-b
# or
prefect worker start --pool my-pool --work-queue queue-a --work-queue queue-b

Notes: - If you omit

--work-queue

entirely, the worker will poll all queues in the pool. - A comma-separated list (e.g.,

-q a,b

) will be treated as a single queue name, not two.

Short answer: you can run multiple flows in parallel on a single Process worker by starting it with a concurrency limit, but concurrent pip installs into the same environment will collide and can crash the worker. That’s almost certainly what you hit. What to know about the Process worker in 3.4.14: - Each flow run is launched as its own subprocess, but they all share the same Python environment as the worker. - Concurrency is controlled by the worker’s limit and by work-queue limits: - Worker-level: start with a cap

prefect worker start -p <pool> -l 4 -q <queue-a> -q <queue-b>

- Queue-level: cap a specific queue

prefect work-queue set-concurrency-limit <queue-name> 1 -p <pool>

- Docs: Workers, Worker CLI, Work pools and queues Why the crash happened: - If your deployments use pull steps that run pip (e.g.,

pip_install_requirements

) and you allow the Process worker to run multiple flow runs at once, those pip installs happen concurrently into the same site-packages and caches. That can cause lock contention, partial installs, or cache corruption, and the worker process may crash. Safe patterns for parallelism with “pip-install” workflows: - Best: switch to an isolated runtime per run - Docker worker: each run gets its own container and can safely install packages

prefect worker start -p <pool> --type docker -l 4

- Or prebuild images with all dependencies; don’t pip install at runtime. - If you must stay on Process worker: - Pre-install all dependencies on the machine/venv before starting the worker, then remove the runtime pip-install steps. After that, you can raise

-l

safely. - Or serialize only the “pip-install” deployments: - Put them on a dedicated work queue with a concurrency limit of 1:

prefect work-queue set-concurrency-limit pip-installs 1 -p <pool>

- Keep other deployments on different queues that can run in parallel. - Last resort: run the Process worker with

--limit 1

so installs never overlap. Advanced option (only if you’re comfortable customizing): - Avoid touching the shared site-packages by installing into a per-run target directory and augmenting

PYTHONPATH

. - Use a custom pull step to: - pip install with

--target .deps

in the run’s working directory - set

PYTHONPATH

to include that path via deployment

job_variables.env

- This keeps installs isolated, but requires careful setup and testing. If you can share: - Your worker start command (including any

-l/--limit

you used) - Snippet of the deployment’s pull steps (especially how pip install is invoked) - A snippet of the crash/error logs from the worker I can help pinpoint whether it’s a pip lock/cache issue vs something else and propose the cleanest change for your setup.

Great question — this behavior is expected with serve(), and here’s why plus how to avoid it. What’s happening -

flow.serve()

starts a local Runner that polls only for the deployments you registered with that serve call. It does not use work pools or work queues at all. - When you pass a deployment with a

work_pool_name

into

serve()

, Prefect ignores the work pool and warns; the deployment is served locally instead. That means scheduled runs for that served deployment will be executed by the serve process, even if there are no workers on its queue. Docs for reference: - Serving flows in local processes: Run flows in local processes - Flow/Runner API: Flow API and Runner API How to avoid serve() picking up those runs Choose one of these patterns: - Don’t serve deployments that you want workers to run - If a deployment should only be executed by workers/queues, do not include it in your

serve()

call in the “regular deployments” folder. - Separate deployments - Use

serve()

only for specific local/test deployments that should run in that folder. - Create separate deployments for worker-driven execution, attached to a work pool/queue, and don’t pass those to

serve()

. - Run serve() in a different workspace or with distinct deployment names - If accidental overlap is possible, use a different Prefect workspace (Org/Project) for your served flows; or give served deployments distinct names/namespaces so you don’t register the same deployment in both places. - If you temporarily need to stop serve() from running anything - Stop the serve process; or remove those deployments from the runner (by stopping serve and not adding them again). - Alternatively, pause those deployments in the UI if you’re just trying to halt local execution. Key takeaway - serve() is a self-contained execution path for the deployments you explicitly register with it; it bypasses work pools/queues. If you see runs executing from your “regular deployments” folder, that’s because those deployments were registered with serve() in that folder. To force runs to require a worker, don’t serve those deployments; create them with a work pool/queue and let the worker pick them up.

There isn’t a built-in “broadcast to all workers” button in Prefect 3.x. Workers pull work from queues; you can’t target a worker by name directly. But you can achieve a “once per worker/machine” rollout with a couple of reliable patterns. Option A: one queue per worker (simple, deterministic) - Create a dedicated queue for each worker/machine. - Start each worker pinned to its queue:

prefect worker start -p my-pool -q worker-1
  prefect worker start -p my-pool -q worker-2
  prefect worker start -p my-pool -q worker-3

- When you need to run your update everywhere, trigger one flow run per queue. - From Python:

from prefect.deployments import run_deployment

    for q in ["worker-1", "worker-2", "worker-3"]:
        run_deployment("update-flow/update-deploy", work_queue_name=q)

- Notes: - You can pass

work_queue_name

via Python (`run_deployment`/`arun_deployment` or the client). The CLI

prefect deployment run

does not expose a work-queue flag. - If the queue doesn’t exist but the deployment has a work pool, Prefect will auto-create the queue within that pool. Option B: tag/label based routing with separate queues - Define queues whose filters/labels map to workers (e.g., worker label, hostname). - Start workers to only poll their matching queue(s). - Trigger one run per targeted queue as above. - This is more flexible if workers are grouped (e.g., region, GPU vs CPU). Option C: coordinator flow that fans out - Write a “broadcast coordinator” that submits N runs, one per queue/worker group. - Track completion/health and surface a single overall status. What won’t work - Targeting a specific worker by name from Prefect: not supported by design. - A single “run once” automatically fanning out to all workers: not built-in. Practical tips - Keep a registry/list of active worker queues (even a small config file or a table you maintain) so your broadcast script can iterate the current set. - Consider a small “agent” flow that runs on each machine’s dedicated queue to perform machine-level updates (e.g., pull code, restart services), keeping your rollout logic consistent. - If you use serve() alongside workers, remember serve() ignores work pools/queues and only executes the deployments it serves; don’t use it for broadcasts targeting workers. Docs for reference: - Workers and pools: Workers and Work pools - Running a deployment from Python (with work_queue_name): Deployments run API