Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi all, is it possible to triage deployments to a work pool dynamically, say, if the first work pool is busy, etc. Either round robin style or anything else? Thanks

<@U021J8TU753> Bump - any help would be appreciated. Thanks!

Hi <@U04TJA37S12> - I don’t think you can specify more than 1 work pool on a deployment

Hi <@U02L949MUGM> ah i see. My concern was wanted to, for example, triage jobs to execute on workers dynamically within the work pool. For example, if worker 1 is busy, execute on worker 2. etc

if using one work pool to programthis, is that possible instead?

you can have multiple workers pooling a single work pool! so if one is unavailable, another should pick up and schedule the work

ah makes sense thank you for the quick response <@U02L949MUGM>! ONe last question, is there a concurrency limit where we will see performance hits when we run lots of deployments? Im on the server version of Prefect. Can I mitigate this by opening up more work pools?

it would depend on the volume of flows but as long as your infrastructure can scale to handle the flows then you shouldn’t have an issue with the worker. you can always scale the worker vertically if you’re hitting memory constraints 