Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

what's the recommended  concurrency limit per work queue and per work pool when using the default task runner?

hi <@U03N0F9JP5J> - I'd say there's no one size fits all answer for concurrency, instead I think its very much a function of your runtime infrastructure and workflow details (e.g. are you hitting an API with some concurrent connection limits etc)

if you could provide some more details on your workflow, perhaps I could offer a general recommendation

<@U02GMEZU18B> thanks! I'm running the agent on a single compute instance with Docker. The agent starts two queues on the default work pool. I've set the default work pool concurrency limit to 20 with no limits on the work queue.

However, I do have a Flow that creates many subflows (&gt; 50) and these subflows are assigned to its own queue. The queue immediately becomes "unhealthy".

Is queue health determined by `num of current runs / num of upcoming runs`?

ah - if this is what I think it is, this is something we're thinking about changing down the road.

I suspect that the flow runs that are submitted after your 20 concurrency slots are already filled are being marked as LATE while waiting for the slots to free, and then work queues are marked unhealthy because of the LATE flow runs

is your work pool concurrency of 20 a hard limit?

<@U02GMEZU18B> no not really but I found that having more than ~30 concurrent runs breaks my agent

<@U03N0F9JP5J> what do you mean by "breaks my agent" in this case? I would anticipate your work queue becomes unhealthy as your runs exceed the concurrency limit and then pile up as LATE

<@U02GMEZU18B> CPU and mem usage reaches 100% even though I have an instance with 4 vCPUs and 16 GB Memory.