Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Scaling question: forgive me if I missed this in the docs/github-issues (I have looked), but do flows executed with ECSRun auto-scale in some way? Ie, do mapped tasks turn into ECS tasks?  Or does some ECS-service/task setting do the job based on resource consumption?  Or do I need a Dask cluster for that?  Does a Dask cluster even do this directly?

The ECSRun will create a task-definition that runs your flow. So mapped tasks will be in the same task definition. I don’t think ECS would provide autoscaling because the autoscaling mechanism is increasing the number of tasks.

I think the easiest way to have a Flow with a DaskExecutor that autoscales would be to use <https://coiled.io/|Coiled> (and they can be backed by AWS)

If you are authenticated with Coiled, you can just do:
```import coiled

executor = DaskExecutor(
    cluster_class=coiled.Cluster,
    cluster_kwargs={
        "software": "jrbourbeau/prefect",
        "shutdown_on_close": False,
        "name": "prefect-executor",
    },
)

flow.run(
    executor=executor,
)```
and this will spin up a Dask cluster hosted by coiled upon flow run. More info <https://docs.coiled.io/user_guide/examples/prefect.html|here> .