Brett Jurman

    Brett Jurman

    1 year ago
    For running prefect with coiled, is there a way to set the resource requirments needed for a task to run (Maybe on a per task basis)? I'm seeing multiple tasks schedule on the same machine and leave the rest idle
    1. I'd like to use the whole cluster regardless. 2. I'd like to be able to say that some tasks have memory requirements so that multiple tasks don't end up scheduling on the same system.
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Brett Jurman, the short answer is no. Doesn’t Coiled have autoscaling? On the Prefect side if you have steps in a pipeline that have varying resource requirements, what users have done is register subflows with different executors and then use a main flow to orchestrate these.
    Brett Jurman

    Brett Jurman

    1 year ago
    ok good to know. we are gonna confirm the scheduling part, but that sounds more like a dask problem if so.
    Kevin Kho

    Kevin Kho

    1 year ago
    So in Prefect if you have 2 mapped tasks, A and B. Let’s say you have 10 workers and A needs 7 and B needs 3. What tends to happen is just A runs and B waits for it to complete.