https://prefect.io logo
c

Clemens

10/29/2020, 3:21 PM
Hi everyone, I am trying to connect prefect to a local dask cluster to test out distributed calculation of a flow that I want to register to a dask cluster in the cloud later. I followed the tutorial https://docs.prefect.io/core/advanced_tutorials/dask-cluster.html and that works great as long as everything is defined in one python file. But if I want to import any function from a second file running it results in a
ModuleNotFoundError
. Does anybody have any experience on how Dask is handling these imports?
z

Zanie

10/29/2020, 3:32 PM
Hi! This kind of depends how you’re storing your flows. Perhaps this recent thread would be helpful! https://prefect-community.slack.com/archives/CL09KU1K7/p1603404858002800?thread_ts=1603396451.488900&cid=CL09KU1K7
upvote 1
c

Clemens

10/29/2020, 3:35 PM
thanks, i will have a look! to answer the question: I haven’t set up any storage. Just running the tutorial file as is and moving one function to another file and then reimporting breaks it.
n

nicholas

10/29/2020, 3:39 PM
That makes sense @Clemens - Prefect doesn't do anything special to package your flow and so relies on imports existing in the runtime environment; dependencies should either be pip-installable or present in the environment. As @Zanie mentioned, this is usually resolved by packaging your flows using whatever storage is appropriate for your setup.
z

Zanie

10/29/2020, 3:40 PM
If storage is not set, it defaults to using LocalStorage as described at https://docs.prefect.io/orchestration/execution/storage_options.html#local
There’s also a stack overflow answer on this https://stackoverflow.com/questions/63881231/prefect-modulenotfounderror-when-running-from-ui — generally the trick is ensuring that your files / module are on the python path while being executed so they can be imported
upvote 1
c

Clemens

10/29/2020, 5:46 PM
got it running, thank’s a lot!!
🚀 2