Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

For running prefect with coiled, is there a way to set the resource requirments needed for a task to run (Maybe on a per task basis)? I'm seeing multiple tasks schedule on the same machine and leave the rest idle

1. I'd like to use the whole cluster regardless.
2. I'd like to be able to say that some tasks have memory requirements so that multiple tasks don't end up scheduling on the same system.

Hey <@U027XH8RALA>, the short answer is no. Doesn’t Coiled have autoscaling? On the Prefect side if you have steps in a pipeline that have varying resource requirements, what users have done is register subflows with different executors and then use a main flow to orchestrate these.

ok good to know. we are gonna confirm the scheduling part, but that sounds more like a dask problem if so.

So in Prefect if you have 2 mapped tasks, A and B. Let’s say you have 10 workers and A needs 7 and B needs 3. What tends to happen is just A runs and B waits for it to complete.