p

    Preston Marshall

    2 years ago
    So I made a custom Secret implementation that pulls from google secret manager. The problem is that the dask-workers blow up because they can't seem to find the module. It's just a file in my demo directory. How is this supposed to work?
    Chris White

    Chris White

    2 years ago
    Hi Preston - because the code you submit to a dask cluster runs on different machines, those machines need all the same importable dependencies as your code. In this case it sounds like your code relies on importing from a file that is not available on your worker machines
    The dask client has some helper methods for pushing files to the workers that you might want to check out, or you could bake the file into your docker image
    p

    Preston Marshall

    2 years ago
    well, I actually have the whole app on an image and am telling the workers to use that image. still though dask-worker seems unable to find modules running in the CWD
    do I have to use preload to add the current directory to the load path?
    Chris White

    Chris White

    2 years ago
    Yea just having the file on the machine isn’t sufficient - it needs to be importable; you could add that directory to your path or push the file to a known importable location
    p

    Preston Marshall

    2 years ago
    ok I'll try that
    Chris White

    Chris White

    2 years ago
    👍
    p

    Preston Marshall

    2 years ago
    --preload did the trick
    import sys
    import os.path
    cwd = os.path.dirname(__file__)
    sys.path.append(cwd)
    Chris White

    Chris White

    2 years ago
    Nice!