https://prefect.io logo
Title
t

Tsang Yong

11/07/2019, 12:10 AM
Is there a recommended way to build workflow code and organize local imports. how should I properly organize the workflow code / utils imports over a dask cluster? I’m new to python and I find myself having to build eggs + upload them while these are not my familiar territory.
c

Chris White

11/07/2019, 12:11 AM
anything that you import locally when you build your flow needs to also be importable at run time on all of the dask workers; typically people use the
upload_file
Dask API to upload their file import dependencies: https://distributed.dask.org/en/latest/api.html#distributed.Client.upload_file
in fact, Matt Rocklin (the creator of Dask) and I are working on a blog post on precisely this topic - I’ll be sure to post it here when it’s available
🤓 1
t

Tsang Yong

11/07/2019, 12:13 AM
Ok. looking forward to the post. thanks!