@Marvin What Fargate cluster does in DaskRunner?

The Fargate cluster in the DaskRunner for Prefect is used to run a temporary Dask cluster on AWS Fargate. This allows you to execute tasks in a distributed environment where the resources can be managed efficiently, scaling as needed. You can configure the

DaskTaskRunner

to use a

FargateCluster

by specifying the cluster class and additional arguments such as the number of workers and the Docker image to use. For example:

python
from prefect.run_configs import DaskTaskRunner

DaskTaskRunner(
    cluster_class="dask_cloudprovider.aws.FargateCluster",
    cluster_kwargs={"n_workers": 4, "image": "my-prefect-image"},
)

In this configuration, the DaskRunner will create a FargateCluster with 4 workers using the specified image when starting a flow run.

Does it spawns new ECS tasks on flow?