Thread
#prefect-community
    s

    Stewart Webb

    3 years ago
    Hi All, I'm trying to figure out what the best-practice would be within Prefect if within a task I need to send a job to a 3rd party appliance which cannot scale, so I need to not send too many concurrent jobs to it. All I can find in relation to this is the queued state I could put a given Task in. I assume then I'd be looking at building my own resource queuing system (using redis-or-the-like) to queue these jobs..
    Chris White

    Chris White

    3 years ago
    Hi Stewart - yea this is a common request; currently the only way to do this in Prefect Core is to use a
    DaskExecutor
    and dask worker resources (see the docs and link here: https://docs.prefect.io/api/unreleased/engine/executors.html#daskexecutor)
    as you’re aware, managing the number of concurrent running tasks requires some sort of state + orchestration layer - this will be a first-class feature in Prefect Cloud but we haven’t figured out a good lightweight way of introducing it into Core — other people have done as you say and hooked up a Redis Queue and then use a Task
    state_handler
    to check the queue prior to entering a Running state
    s

    Stewart Webb

    3 years ago
    Thanks very much for this response, chris