https://prefect.io logo
m

Michael Hadorn

11/16/2020, 8:31 AM
Hi there Have a question about the use of the dask executor with own python modules. If I have to use own python modules, i got this exception:
Copy code
[2020-11-16 09:26:55+0100] ERROR - prefect.Load | Unexpected error occured in FlowRunner: ModuleNotFoundError("No module named 'xxx'")
I think I have to upload all my necessary files via:
client.upload_file('xxx.py')
(see https://distributed.dask.org/en/latest/api.html#distributed.Client.upload_file) But how can i access the dask client? Here the client would be available while the run: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/engine/executors/dask.py#L257 But directly after, it's cleared again. Does anybody have a hint for me? Regards Michael
The same question also here: https://github.com/PrefectHQ/prefect/discussions/3670 (sorry i'm not sure where to post...
z

Zanie

11/16/2020, 3:19 PM
Hi! You can just make sure your module is installed in the environment where you run your dask worker.
upvote 1
Otherwise, you can make use of Flow Storage (https://docs.prefect.io/api/latest/environments/storage.html) like our Docker storage and install your module
m

Michael Hadorn

11/18/2020, 8:22 AM
@Zanie Hi! Thanks for the answer. If this files are only .py files written by me. Not a full module. Storage seems me too much (go over git or so). Maybe do you have an example? In my case, i do only have some mini classes for managing objects (and i would like to avoid to have all the code in one file). So I only need to add these files to dask.
z

Zanie

11/18/2020, 4:00 PM
Storage is used no matter what, the default is just to use local storage where your flow is pickled.
The issue here is that you do have a Python “module” with some utilities
xxx
and that is not being provided at runtime.
If you make sure that
xxx
is available on the PYTHON_PATH where your flow is run then the utilities will import correctly
m

Michael Hadorn

11/18/2020, 8:53 PM
ok. thanks i will try again. best, michael
4 Views