https://prefect.io
Join Slack
Menu
Home
Archive
Hi all, I would like to limit the number of flows that a worker may run, for a specific flow. i.e. I...
# best-practices
j
Hi all, I would like to limit the number of flows that a worker may run, for a specific flow. i.e. I'd like each worker to run a maximum of 10 concurrent flows for flow x, while still being able run more flows for flow y
My first thought was to use:
Copy code
async with concurrency(host, occupy=1):
But that doesn't stop a worker picking up the flow to begin with. Which means it could pick up 20 flows and block on 10. Ideally those other 10 would be picked up by another worker as soon as it's available, instead of being blocked on the first worker.
Any suggestions?
d
Can you use a separate work pool for flow x?
j
@Dev Dabke not sure that would help. Can a worker run in multiple work pools? Also, it seems that the concurrency limit in a work pool limits the maximum number of flows for the whole pool, not per worker.
d
I believe a worker is restricted to a single pool, but I believe different work queues in the same work pool can have different concurrency limits
image.png
You could create a queue just for the one flow and set a specific concurrency limit for that, perhaps?
I believe workers can pull from multiple work queues in the same work pool
j
Yea but again the limit for the queue will be across all workers... I'm essentially trying to ensure the workers are load balanced, but what I find is one worker hoards half the flows and others are doing nothing. So ideally if I know that a worker can comfortably handle say 10 flows at a time, I want to limit each worker to 10, while not limiting how many a queue or pool can run across workers
It looks like the only way is to start the worker with a limit, but that will apply to all flows types.
52 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by