Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi All, I'm trying to figure out what the best-practice would be within Prefect if within a task I need to send a job to a 3rd party appliance which cannot scale, so I need to not send too many concurrent jobs to it. All I can find in relation to this is the queued state I could put a given Task in. I assume then I'd be looking at building my own resource queuing system (using redis-or-the-like) to queue these jobs..

Hi Stewart - yea this is a common request; currently the only way to do this in Prefect Core is to use a `DaskExecutor` and dask worker resources (see the docs and link here: <https://docs.prefect.io/api/unreleased/engine/executors.html#daskexecutor>)

as you’re aware, managing the number of concurrent running tasks requires some sort of state + orchestration layer - this will be a first-class feature in Prefect Cloud but we haven’t figured out a good lightweight way of introducing it into Core — other people have done as you say and hooked up a Redis Queue and then use a Task `state_handler` to check the queue prior to entering a Running state

Thanks very much for this response, chris