Is there a recommended way to build workflow code and organize local imports. how should I properly organize the workflow code / utils imports over a dask cluster? I’m new to python and I find myself having to build eggs + upload them while these are not my familiar territory.
c
Chris White
11/07/2019, 12:11 AM
anything that you import locally when you build your flow needs to also be importable at run time on all of the dask workers; typically people use the
in fact, Matt Rocklin (the creator of Dask) and I are working on a blog post on precisely this topic - I’ll be sure to post it here when it’s available