Hi All, I'm trying to figure out what the best-pra...
# prefect-community
Hi All, I'm trying to figure out what the best-practice would be within Prefect if within a task I need to send a job to a 3rd party appliance which cannot scale, so I need to not send too many concurrent jobs to it. All I can find in relation to this is the queued state I could put a given Task in. I assume then I'd be looking at building my own resource queuing system (using redis-or-the-like) to queue these jobs..
Hi Stewart - yea this is a common request; currently the only way to do this in Prefect Core is to use a
and dask worker resources (see the docs and link here: https://docs.prefect.io/api/unreleased/engine/executors.html#daskexecutor)
as you’re aware, managing the number of concurrent running tasks requires some sort of state + orchestration layer - this will be a first-class feature in Prefect Cloud but we haven’t figured out a good lightweight way of introducing it into Core — other people have done as you say and hooked up a Redis Queue and then use a Task
to check the queue prior to entering a Running state
Thanks very much for this response, chris