If I want to run my flows on the ECS fargate, do I setup a DaskExecutor with Fargate with a cluster class? Is it fine if I have an agent local and not using the ECS agent?
What's the difference between setting an
vs for example changing the
02/21/2022, 4:57 AM
I think you can mix and match executor and RunConfig and it should work. Anyway, it would spin up the DaskExecutor as a new Fargate task whether the RunConfig is LocalRun or ECSRun. Does that answer you?
I think the answer is there is no difference because it’s a new process either way
02/21/2022, 5:09 AM
@Kevin Kho If I want to distribute my flows, meaning different dags gets executed on different fargate instances, what do I need to do?
02/21/2022, 5:19 AM
DaskExecutor using FargateCluster. But if you really want one task = one fargate instance, then it’s probably better as subflows that have ECSRun configured on them
I made a mistake in the earlier state. It should be “it would spin up the DaskExecutor as a new ECS cluster” per this
02/21/2022, 5:23 AM
It doesn't have to one task = one farget, but having N instances that can be set somewhere. For example if I set 3 instances, then it would be at most 3 instances, and then shutoff once the task is done.
I'll check out the link and DaskExecutor, thank you.
02/21/2022, 5:25 AM
If you have a flow and 1 specific task needs the cluster, you can use the ResourceManager as well so you really control the spin down.
But the DaskExecutor should shut down after Flow execution too