@Marvin Does FargateCluster creates new ECS Tasks on Fargate when used in DaskTaskRunner?

Yes, the

FargateCluster

within the

DaskTaskRunner

does create new ECS tasks on Fargate when used with Dask in Prefect. In the context of Prefect, Dask clusters can be created to run multiple tasks in parallel. There was an issue opened in Prefect's GitHub repository showing the use of

FargateCluster

with

DaskTaskRunner

. Here's an example, gleaned from that issue, of how

FargateCluster

was adapted for `DaskTaskRunner`:

python
from prefect import task, flow
from prefect.task_runners import DaskTaskRunner

fargate_dask_runner = DaskTaskRunner(
    cluster_class="dask_cloudprovider.aws.FargateCluster",
    adapt_kwargs={
        "maximum": 100,
    }
)

@task
def say_hello(name):
    print(f"hello {name}")

@task
def say_goodbye(name):
    print(f"goodbye {name}")

@flow(task_runner=fargate_dask_runner)
def greetings(names):
    for name in names:
        say_hello(name)
        say_goodbye(name)

if __name__ == "__main__":
    greetings(["arthur", "trillian", "ford", "marvin"])

In this example, a Dask cluster backed by AWS Fargate is used to run tasks in parallel within a flow. The

FargateCluster

should be responsible for creating tasks on Fargate. There were some issues reported regarding version mismatches and potentially other problems, but the general approach of using

FargateCluster

with Dask in Prefect is to create ECS Fargate tasks.

how to validate, that new ECS tasks are created?