https://prefect.io
Join Slack
Scaling question: forgive me if I missed this in the docs/github-issues (I have looked), but do flow...
# ask-community
m
Scaling question: forgive me if I missed this in the docs/github-issues (I have looked), but do flows executed with ECSRun auto-scale in some way? Ie, do mapped tasks turn into ECS tasks? Or does some ECS-service/task setting do the job based on resource consumption? Or do I need a Dask cluster for that? Does a Dask cluster even do this directly?
k
The ECSRun will create a task-definition that runs your flow. So mapped tasks will be in the same task definition. I don’t think ECS would provide autoscaling because the autoscaling mechanism is increasing the number of tasks. I think the easiest way to have a Flow with a DaskExecutor that autoscales would be to use Coiled (and they can be backed by AWS)
If you are authenticated with Coiled, you can just do:
Copy code
import coiled

executor = DaskExecutor(
    cluster_class=coiled.Cluster,
    cluster_kwargs={
        "software": "jrbourbeau/prefect",
        "shutdown_on_close": False,
        "name": "prefect-executor",
    },
)

flow.run(
    executor=executor,
)
and this will spin up a Dask cluster hosted by coiled upon flow run. More info here .
m
Ok- thanks Kevin!
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by