Channels
#prefect-community
Title
p
Preston Marshall
02/08/2020, 4:46 PMSo I made a custom Secret implementation that pulls from google secret manager. The problem is that the dask-workers blow up because they can't seem to find the module. It's just a file in my demo directory. How is this supposed to work?
c
Chris White
02/08/2020, 4:52 PM
Hi Preston - because the code you submit to a dask cluster runs on different machines, those machines need all the same importable dependencies as your code. In this case it sounds like your code relies on importing from a file that is not available on your worker machines
The dask client has some helper methods for pushing files to the workers that you might want to check out, or you could bake the file into your docker image
p
Preston Marshall
02/08/2020, 4:53 PMwell, I actually have the whole app on an image and am telling the workers to use that image. still though dask-worker seems unable to find modules running in the CWD
do I have to use preload to add the current directory to the load path?
c
Chris White
02/08/2020, 4:54 PM
Yea just having the file on the machine isn’t sufficient - it needs to be importable; you could add that directory to your path or push the file to a known importable location
p
Preston Marshall
02/08/2020, 4:54 PMok I'll try that
c
Chris White
02/08/2020, 4:55 PM
👍
p
Preston Marshall
02/08/2020, 5:06 PM--preload did the trick
Copy code
import sys
import os.path
cwd = os.path.dirname(__file__)
sys.path.append(cwd)c
Chris White
02/08/2020, 5:06 PM
Nice!