So I made a custom Secret implementation that pulls from goo

Question

So I made a custom Secret implementation that pulls from google secret manager The problem is that the dask workers blow up because they can t seem to find the module It s just a file in my demo directory How is this supposed to work

Chris White · Answer

Hi Preston because the code you submit to a dask cluster runs on different machines those machines need all the same importable dependencies as your code In this case it sounds like your code relies on importing from a file that is not available on your worker machines

Chris White · Answer

The dask client has some helper methods for pushing files to the workers that you might want to check out or you could bake the file into your docker image

Preston Marshall · Answer

well I actually have the whole app on an image and am telling the workers to use that image still though dask worker seems unable to find modules running in the CWD

Preston Marshall · Answer

do I have to use preload to add the current directory to the load path

Chris White · Answer

Yea just having the file on the machine isn t sufficient it needs to be importable you could add that directory to your path or push the file to a known importable location

Preston Marshall · Answer

ok I ll try that

Chris White · Answer

+1

Preston Marshall · Answer

preload did the trick

Preston Marshall · Answer

```import sys import os path cwd = os path dirname file sys path append cwd ```

Chris White · Answer

Nice