So I made a custom Secret implementation that pull...
# ask-communityp
Preston Marshall
02/08/2020, 4:46 PMSo I made a custom Secret implementation that pulls from google secret manager. The problem is that the dask-workers blow up because they can't seem to find the module. It's just a file in my demo directory. How is this supposed to work?
c
Chris White
02/08/2020, 4:52 PM
Hi Preston - because the code you submit to a dask cluster runs on different machines, those machines need all the same importable dependencies as your code. In this case it sounds like your code relies on importing from a file that is not available on your worker machines
Chris White
02/08/2020, 4:52 PM
The dask client has some helper methods for pushing files to the workers that you might want to check out, or you could bake the file into your docker image
p
Preston Marshall
02/08/2020, 4:53 PMwell, I actually have the whole app on an image and am telling the workers to use that image. still though dask-worker seems unable to find modules running in the CWD
Preston Marshall
02/08/2020, 4:53 PMdo I have to use preload to add the current directory to the load path?
c
Chris White
02/08/2020, 4:54 PM
Yea just having the file on the machine isn’t sufficient - it needs to be importable; you could add that directory to your path or push the file to a known importable location
p
Preston Marshall
02/08/2020, 4:54 PMok I'll try that
c
Chris White
02/08/2020, 4:55 PM
👍
p
Preston Marshall
02/08/2020, 5:06 PM--preload did the trick
Preston Marshall
02/08/2020, 5:06 PM Copy code
import sys
import os.path
cwd = os.path.dirname(__file__)
sys.path.append(cwd)c
Chris White
02/08/2020, 5:06 PM
Nice!