If I want to run my flows on the ECS fargate do I setup a Da

Question

If I want to run my flows on the ECS fargate do I setup a DaskExecutor with Fargate with a cluster class Is it fine if I have an agent local and not using the ECS agent

Max Lei · Answer

What s the difference between setting an `ECSRun` vs for example changing the `executor` on `LocalRun`

Kevin Kho · Answer

I think you can mix and match executor and RunConfig and it should work Anyway it would spin up the DaskExecutor as a new Fargate task whether the RunConfig is LocalRun or ECSRun Does that answer you

Kevin Kho · Answer

I think the answer is there is no difference because it s a new process either way

Max Lei · Answer

< Kevin Kho> If I want to distribute my flows meaning different dags gets executed on different fargate instances what do I need to do `ECSRun` or a `DaskExecutor`

Kevin Kho · Answer

DaskExecutor using FargateCluster But if you really want one task = one fargate instance then it s probably better as subflows that have ECSRun configured on them

Kevin Kho · Answer

I made a mistake in the earlier state It should be it would spin up the DaskExecutor as a new ECS cluster per <https cloudprovider dask org en latest aws html fargate|this>

Max Lei · Answer

It doesn t have to one task = one farget but having N instances that can be set somewhere For example if I set 3 instances then it would be at most 3 instances and then shutoff once the task is done

Max Lei · Answer

I ll check out the link and DaskExecutor thank you

Kevin Kho · Answer

If you have a flow and 1 specific task needs the cluster you can use the <https docs prefect io core idioms resource manager html|ResourceManager> as well so you really control the spin down

Kevin Kho · Answer

But the DaskExecutor should shut down after Flow execution too

Max Lei · Answer

Thanks a lot I ll see if I can set one up