Tsang Yong

11/07/2019, 12:10 AM
Is there a recommended way to build workflow code and organize local imports. how should I properly organize the workflow code / utils imports over a dask cluster? I’m new to python and I find myself having to build eggs + upload them while these are not my familiar territory.

Chris White

11/07/2019, 12:11 AM
anything that you import locally when you build your flow needs to also be importable at run time on all of the dask workers; typically people use the
Dask API to upload their file import dependencies:
in fact, Matt Rocklin (the creator of Dask) and I are working on a blog post on precisely this topic - I’ll be sure to post it here when it’s available
🤓 1

Tsang Yong

11/07/2019, 12:13 AM
Ok. looking forward to the post. thanks!