Another conceptual question: how tightly is Prefect bound to Dask? It appears (in the docs) that some critical features are only available when using Dask: for example Task parallelism and transparent data flow between Tasks (which may be running on different machines). We are planning to run everything on ECS, will we still be able to use these features? Thanks a lot!
s
Samuel Hinton
02/23/2021, 12:59 PM
We have our services deployed on ECS and theres no issue with Dask in there. The local dask executor works perfectly
f
Fina Silva-Santisteban
02/23/2021, 5:08 PM
@Tobias Heintz we’re using ECS and the local dask executor as well! The local one is enough to enable parallelism.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.