https://prefect.io
Join Slack
Hi there Have a question about the use of the dask executor with own python modules. If I have to us...
# ask-community
m
Hi there Have a question about the use of the dask executor with own python modules. If I have to use own python modules, i got this exception:
Copy code
[2020-11-16 09:26:55+0100] ERROR - prefect.Load | Unexpected error occured in FlowRunner: ModuleNotFoundError("No module named 'xxx'")
I think I have to upload all my necessary files via: 
client.upload_file('xxx.py')
(see https://distributed.dask.org/en/latest/api.html#distributed.Client.upload_file) But how can i access the dask client? Here the client would be available while the run: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/engine/executors/dask.py#L257 But directly after, it's cleared again. Does anybody have a hint for me? Regards Michael
The same question also here: https://github.com/PrefectHQ/prefect/discussions/3670 (sorry i'm not sure where to post...
z
Hi! You can just make sure your module is installed in the environment where you run your dask worker.
upvote 1
Otherwise, you can make use of Flow Storage (https://docs.prefect.io/api/latest/environments/storage.html) like our Docker storage and install your module
m
@Zanie Hi! Thanks for the answer. If this files are only .py files written by me. Not a full module. Storage seems me too much (go over git or so). Maybe do you have an example? In my case, i do only have some mini classes for managing objects (and i would like to avoid to have all the code in one file). So I only need to add these files to dask.
z
Storage is used no matter what, the default is just to use local storage where your flow is pickled.
The issue here is that you do have a Python “module” with some utilities 
xxx
and that is not being provided at runtime.
If you make sure that 
xxx
is available on the PYTHON_PATH where your flow is run then the utilities will import correctly
m
ok. thanks i will try again. best, michael
8 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by