Stewart Webb

09/15/2019, 12:24 PM
Hi All, I'm trying to figure out what the best-practice would be within Prefect if within a task I need to send a job to a 3rd party appliance which cannot scale, so I need to not send too many concurrent jobs to it. All I can find in relation to this is the queued state I could put a given Task in. I assume then I'd be looking at building my own resource queuing system (using redis-or-the-like) to queue these jobs..

Chris White

09/15/2019, 9:53 PM
Hi Stewart - yea this is a common request; currently the only way to do this in Prefect Core is to use a
and dask worker resources (see the docs and link here:
as you’re aware, managing the number of concurrent running tasks requires some sort of state + orchestration layer - this will be a first-class feature in Prefect Cloud but we haven’t figured out a good lightweight way of introducing it into Core — other people have done as you say and hooked up a Redis Queue and then use a Task
to check the queue prior to entering a Running state

Stewart Webb

09/17/2019, 2:57 PM
Thanks very much for this response, chris