<@ULVA73B9P> What is a good limit to set on my pro...
# ask-marvin
t
@Marvin What is a good limit to set on my process worker? I experienced some performance impact where I wasn't able to access the UI on webpage on my single process worker running on the same machine where my prefect server is hosted. It is all basic operations like database queries and upsert.
m
thought for 347.8 seconds
Short answer: start low. On a box that’s running both Prefect Server and your worker, set the process worker to 1–2 concurrent flow runs and tune up only if the machine stays responsive. Why this happens - A process worker starts each flow run in its own process. If you don’t set limits, it can start many flows concurrently, competing with the Server/UI for CPU and memory. - DB-heavy flows can also saturate DB connections or I/O, making the UI feel unresponsive. How to limit concurrency (Prefect 3.x) - Per-worker limit (requires restart to change):
Copy code
prefect worker start -p <your-pool> --limit 1
or in code:
Copy code
from prefect.workers.process import ProcessWorker
  ProcessWorker(work_pool_name="<your-pool>", limit=1).start()
- Pool-wide limit (affects all workers in the pool; no worker restart needed):
Copy code
prefect work-pool set-concurrency-limit <your-pool> 1
  prefect work-pool inspect <your-pool>
Recommended starting points - If the UI was becoming inaccessible, set both the worker limit and the work-pool limit to 1. Verify the Server/UI stays responsive. - If stable, try 2. On a machine with 4+ CPU cores and plenty of RAM, 2–3 can be okay for IO-bound DB work—monitor CPU, memory, and DB connection usage and only increase if you have headroom. - Keep the pool limit aligned with your per-worker limit (or slightly lower if you might add more workers later), so you don’t accidentally oversubscribe. Optional: throttle specific operations - If certain tasks (e.g., upserts) should never run in parallel across flows, use tag-based concurrency limits:
Copy code
prefect concurrency-limit create db-writes 1
Then tag those tasks/flows with
db-writes
. Docs: Global concurrency limits Docs and references - Workers concept: Workers - Work pools and pool concurrency: prefect work-pool CLI - Process worker API (shows
limit
): ProcessWorker If you share your machine specs (CPU cores/RAM) and target throughput, I can suggest a more precise limit.