Hello, what's the best way to parallelize tasks on ECS using Prefect? Let's say that I have a flow where I list a bunch of files and I want to transform each of them on it's own ECS Task at the same time. Should I still need the DaskRunner to do that?
i
Ivan
07/24/2023, 11:55 AM
I will try to create a deployment for this flow per each file using worker with a type ecstask / ecstask push
Or just create an async logic in your flow but you will also need ECSTask block to process each file in a separate ECSTask.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.