Hi All I m trying to figure out what the best practice would

Question

Hi All I m trying to figure out what the best practice would be within Prefect if within a task I need to send a job to a 3rd party appliance which cannot scale so I need to not send too many concurrent jobs to it All I can find in relation to this is the queued state I could put a given Task in I assume then I d be looking at building my own resource queuing system using redis or the like to queue these jobs

Chris White · Answer

Hi Stewart yea this is a common request currently the only way to do this in Prefect Core is to use a `DaskExecutor` and dask worker resources see the docs and link here <https docs prefect io api unreleased engine executors html daskexecutor>

Chris White · Answer

as you re aware managing the number of concurrent running tasks requires some sort of state + orchestration layer this will be a first class feature in Prefect Cloud but we haven t figured out a good lightweight way of introducing it into Core other people have done as you say and hooked up a Redis Queue and then use a Task `state handler` to check the queue prior to entering a Running state

Stewart Webb · Answer

Thanks very much for this response chris