<@ULVA73B9P> I have a woeker setup in systemd on o...
# ask-marvin
b
@Marvin I have a woeker setup in systemd on one server and prefect server setup on another server. My question is how many jobs that particular worker can process in parallel ? or should I add more workers ?
m
In Prefect 3.x, the number of jobs a worker can process in parallel is determined by the worker's configuration and the infrastructure it's running on. Here's a summary of the relevant information: 1. Concurrency Management: Each work pool can optionally restrict concurrent runs of matching flows. For example, a work pool with a concurrency limit of 5 only releases new work if fewer than five matching runs are in a Running or Pending state. This means you can control how many tasks run concurrently by setting the concurrency limit in the work pool configuration. Learn more about work pools and concurrency. 2. Worker Types: Prefect supports different worker types, such as process, Kubernetes, Docker, ECS, etc. Each type has its own way of handling concurrency. For instance, a process worker executes flow runs in subprocesses, whereas a Kubernetes worker executes them as Kubernetes jobs. The choice of worker type will affect how concurrency is managed. Learn more about worker types. 3. Scaling Workers: If a single worker is not sufficient to handle the desired number of concurrent tasks, you can add more workers to the work pool. This can be done by starting additional instances of the worker, which will then distribute the workload across all available workers. In summary, you can manage the number of concurrent jobs by configuring the concurrency limits in your work pool and possibly scaling out by adding more workers if needed.