Hey,
I need some assistance with a flow I've been working on. The flow involves running multiple tasks in parallel using Dask as the task runner. However, I've encountered an issue where I am unable to import modules within the tasks running on the workers.
To provide more context, I can import the required modules without any problems in my main flow. The problem seems to be isolated to the tasks that are running on the workers. I suspect this might be related to the worker's environment or some configuration issue.
Any insights or suggestions on how to resolve this issue would be greatly appreciated!
Thanks in advance for your help.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.