Hi there Have a question about the use of the dask...
# prefect-community
Hi there Have a question about the use of the dask executor with own python modules. If I have to use own python modules, i got this exception:
Copy code
[2020-11-16 09:26:55+0100] ERROR - prefect.Load | Unexpected error occured in FlowRunner: ModuleNotFoundError("No module named 'xxx'")
I think I have to upload all my necessary files via:
(see https://distributed.dask.org/en/latest/api.html#distributed.Client.upload_file) But how can i access the dask client? Here the client would be available while the run: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/engine/executors/dask.py#L257 But directly after, it's cleared again. Does anybody have a hint for me? Regards Michael
The same question also here: https://github.com/PrefectHQ/prefect/discussions/3670 (sorry i'm not sure where to post...
Hi! You can just make sure your module is installed in the environment where you run your dask worker.
upvote 1
Otherwise, you can make use of Flow Storage (https://docs.prefect.io/api/latest/environments/storage.html) like our Docker storage and install your module
@Zanie Hi! Thanks for the answer. If this files are only .py files written by me. Not a full module. Storage seems me too much (go over git or so). Maybe do you have an example? In my case, i do only have some mini classes for managing objects (and i would like to avoid to have all the code in one file). So I only need to add these files to dask.
Storage is used no matter what, the default is just to use local storage where your flow is pickled.
The issue here is that you do have a Python “module” with some utilities
and that is not being provided at runtime.
If you make sure that
is available on the PYTHON_PATH where your flow is run then the utilities will import correctly
ok. thanks i will try again. best, michael