For running prefect with coiled, is there a way to set the resource requirments needed for a task to run (Maybe on a per task basis)? I'm seeing multiple tasks schedule on the same machine and leave the rest idle

Hey @Brett Jurman, the short answer is no. Doesn’t Coiled have autoscaling? On the Prefect side if you have steps in a pipeline that have varying resource requirements, what users have done is register subflows with different executors and then use a main flow to orchestrate these.

ok good to know. we are gonna confirm the scheduling part, but that sounds more like a dask problem if so.

So in Prefect if you have 2 mapped tasks, A and B. Let’s say you have 10 workers and A needs 7 and B needs 3. What tends to happen is just A runs and B waits for it to complete.