@Marvin can I configure a concurrency limit at a worker level so that the worker is not overloaded ?

Yes, you can configure concurrency limits at the worker level in Prefect to prevent overloading. This is typically managed through work pools or work queues, which can restrict concurrent runs of flows. Here are some key points: 1. Concurrency Limits at Work Pool Level: You can set concurrency limits for work pools. For example, if a work pool has a concurrency limit of 5, it will only release new work if fewer than five matching runs are in a "Running" or "Pending" state. This helps manage the workload distributed to workers. 2. Setting Limits: You can set these limits using the Prefect CLI, Prefect API, or Prefect Cloud UI. Commands like

prefect work-pool pause

or

prefect work-pool resume

can control the flow of work to the workers. 3. Global Concurrency Limits: These can be configured with slot decay to control how often tasks can run. The decay rate determines how quickly slots refresh, allowing for a balance between task frequency and system load. For more detailed guidance, you can refer to the following links: - Concurrency Limits - Work Pools If you're using Prefect 3.x, these configurations should be applicable. If you need more specific examples or have other questions, feel free to ask.

@Marvin this is setting at a work pool, but I want to set it at a worker level

Unfortunately, it seems that Prefect currently supports configuring concurrency limits at the work pool, work queue, and deployment levels, but not directly at the individual worker level. The system is designed to manage concurrency through these broader structures rather than on a per-worker basis. You can still manage task concurrency by setting limits on work pools or queues, which indirectly affects the workload that a worker can receive. For more detailed configurations, you can explore using tags to set task run concurrency limits. For further exploration, you may refer to: - Task Run Limits - Work Pools If worker-level concurrency becomes a necessity, you might consider submitting a feature request to the Prefect team or checking their GitHub repository for related discussions.

Yeah I would look into settings for the Task Runners offered. I am using the DaskTaskRunner and I have set

n_workers

to be a fraction of the number of cores available,

threads_per_worker

to be 1, and

memory_limit='auto'

on the

cluster_kwargs

.