Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hey guys, a quick question: can flows defined by the `ECSRun` configuration be ran in a threaded mode, similar how `LocalDaskExecutor`  has the `scheduled=threads` configuration?

So far I was unable to use the `map` functionality of a Prefect `task` in order to have multiple tasks executed in parallel.

Hey <@U020BC7KNQ5>! We’re still discussing the question about cpu and memory by the way. We’ll see if we need to open an issue today. For this one, you should be able to use `LocalDaskExecutor` with `ECSRun` , but if this is a follow up from yesterday, this might be because you’re still getting 1 cpu?

I'll try to configure `LocalDaskExecutor` with `ECSRun`.

Regarding the CPU/memory question from yesterday - I've tried running a task and logging the CPU/RAM available from within the task via script, but it seems it returns the physical infrastructure properties of the AWS machine it's running on - not the limits of the Docker container (for example the task definition had 1CPU/2GB RAM, and the output showed it had 2 CPU/8GB RAM - this is without me trying to override any configuration).

I've managed to find a workaround by specifying the cpu and memory in task definition, and using that one instead of the default one (the task definition in AWS ECS now shows to correct CPU/RAM). However, I would want to try to avoid creating specific task definitions.

Btw, even if using the 1 CPU on the task, I should still be able to use threaded executor, right?

<@U01QEJ9PP53> so... I've added the `LocalDaskExecutor` to my flow configuration, that has the `ECSRun` run configuration already, and now the resource allocation works - ECS tasks have 2CPU/4GB as I set in the `ECSRun` parameters.

I was not aware that setting the executor might affect this behaviour, but the good news is - it works now (as well as the threaded execution) :slightly_smiling_face:

Oh ok good to know everything you mentioned. We’ll update you on that ECSRun cpu and memory situation.