Hey has anyone had any experience setting up a DaskTaskRunner for fargate on a subflow? I’m currently trying to use the default task runner for my parent flow but dask for a subflow and am experiencing some weird behavior once running parallel tasks. I’m not sure if this is related to dask or how I am setting up the task runner. I can confirm that behavior is normal when configuring the dask task runner for the parent flow and not the subflow.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
