If I want to run my flows on the ECS fargate, do I...
# ask-communitym
Max Lei
02/21/2022, 4:39 AMIf I want to run my flows on the ECS fargate, do I setup a DaskExecutor with Fargate with a cluster class? Is it fine if I have an agent local and not using the ECS agent?
Max Lei
02/21/2022, 4:41 AMWhat's the difference between setting an
ECSRunexecutorLocalRunk
Kevin Kho
02/21/2022, 4:57 AM
I think you can mix and match executor and RunConfig and it should work. Anyway, it would spin up the DaskExecutor as a new Fargate task whether the RunConfig is LocalRun or ECSRun. Does that answer you?
Kevin Kho
02/21/2022, 4:57 AM
I think the answer is there is no difference because it’s a new process either way
m
Max Lei
02/21/2022, 5:09 AM@Kevin Kho If I want to distribute my flows, meaning different dags gets executed on different fargate instances, what do I need to do?
ECSRunDaskExecutork
Kevin Kho
02/21/2022, 5:19 AM
DaskExecutor using FargateCluster. But if you really want one task = one fargate instance, then it’s probably better as subflows that have ECSRun configured on them
Kevin Kho
02/21/2022, 5:22 AM
I made a mistake in the earlier state. It should be “it would spin up the DaskExecutor as a new ECS cluster” per this
m
Max Lei
02/21/2022, 5:23 AMIt doesn't have to one task = one farget, but having N instances that can be set somewhere. For example if I set 3 instances, then it would be at most 3 instances, and then shutoff once the task is done.
Max Lei
02/21/2022, 5:24 AMI'll check out the link and DaskExecutor, thank you.
k
Kevin Kho
02/21/2022, 5:25 AM
If you have a flow and 1 specific task needs the cluster, you can use the ResourceManager as well so you really control the spin down.
Kevin Kho
02/21/2022, 5:25 AM
But the DaskExecutor should shut down after Flow execution too
m
Max Lei
02/21/2022, 5:28 AMThanks a lot, I'll see if I can set one up.
👍 1
3 Views