<@ULVA73B9P> can i move workers (process) to a sep...
# ask-community
i
@Marvin can i move workers (process) to a separate machine ? And scale them behind a load balancer
m
thought for 415.2 seconds
Short answer: Yes, you can run process workers on other machines, and you don’t need (or use) a load balancer for them. Details: - Process workers are just lightweight processes that poll the Prefect API and launch your flow runs as local subprocesses. They do not accept inbound traffic, so there’s nothing to “put behind” a load balancer. Scaling is done by running more workers and controlling concurrency. - You can run as many workers as you want on any machines that: - Can reach your Prefect API (Cloud or self-hosted) - Can access your flow code/artifacts (via your deployment’s source) - Have the required Python environment/dependencies How to do it (Prefect 3.x): 1) Create a process work pool (once)
Copy code
prefect work-pool create my-process-pool --type process
2) Start workers on separate machines (point them at the same pool and optional queues)
Copy code
# Machine A
prefect worker start -p my-process-pool -q default

# Machine B
prefect worker start -p my-process-pool -q default
Each worker will poll for work and “pull” flow runs—this provides natural load distribution without a load balancer. 3) Control concurrency - Per-pool limit (caps total concurrent runs across all workers in the pool):
Copy code
prefect work-pool set-concurrency-limit my-process-pool 10
- Per-queue limit (caps concurrent runs for a specific queue in that pool):
Copy code
prefect work-queue set-concurrency-limit <your-queue-name> 5
- Per-worker limit: you can cap how many runs a single worker will execute concurrently (useful when you know a machine’s capacity). Notes on load balancers: - Not needed for process workers—they initiate outbound polling to Prefect and don’t expose an HTTP service. - If you are self-hosting the Prefect API/Server, you can absolutely put the API behind a load balancer, but that’s separate from the workers themselves. Docs and references: - Workers concepts - Process worker API - Global concurrency limits - Background: Work pools and workers blog If you share a bit about your environment (Prefect Cloud vs. self-hosted, OS, and expected concurrency), I can suggest concrete settings and a rollout plan for your workers.