https://prefect.io logo
Title
s

Stewart Webb

09/15/2019, 12:24 PM
Hi All, I'm trying to figure out what the best-practice would be within Prefect if within a task I need to send a job to a 3rd party appliance which cannot scale, so I need to not send too many concurrent jobs to it. All I can find in relation to this is the queued state I could put a given Task in. I assume then I'd be looking at building my own resource queuing system (using redis-or-the-like) to queue these jobs..
c

Chris White

09/15/2019, 9:53 PM
Hi Stewart - yea this is a common request; currently the only way to do this in Prefect Core is to use a
DaskExecutor
and dask worker resources (see the docs and link here: https://docs.prefect.io/api/unreleased/engine/executors.html#daskexecutor)
as you’re aware, managing the number of concurrent running tasks requires some sort of state + orchestration layer - this will be a first-class feature in Prefect Cloud but we haven’t figured out a good lightweight way of introducing it into Core — other people have done as you say and hooked up a Redis Queue and then use a Task
state_handler
to check the queue prior to entering a Running state
s

Stewart Webb

09/17/2019, 2:57 PM
Thanks very much for this response, chris