l

    Lew Dawson

    11 months ago
    Afternoon folks. Wondering if anyone has advice. Struggling to get a simple/common Python pattern working when using
    DockerRun
    run_config with
    Github
    storage. Have the following project structure in the Github repo:
    etl/
    ├── util/
    │   ├── __init__.py
    │   ├── util_vault.py
    ├── __init__.py
    ├── runner.py
    The
    util_vault.py
    looks like:
    from prefect.client import Secret
    
    class UtilVault:
    
        @staticmethod
        def get_secret_simple(name):
            return Secret(name).get()
    The
    runner.py
    looks like:
    from prefect import Flow, task, context
    from prefect.run_configs import DockerRun
    from prefect.storage import GitHub
    
    from util.util_vault import UtilVault
    
    logger = context.get("logger")
    
    
    @task
    def hello_secrets():
        secret = UtilVault.get_secret_simple('GITHUB_ACCESS_TOKEN')
        <http://logger.info|logger.info>('secret = %s', secret)
    
    
    with Flow(name='hello_world') as flow:
        hello_secrets()
    
    flow.storage = GitHub(
        repo='<username>/<repo>',
        path='etl/runner.py',
        access_token_secret='GITHUB_ACCESS_TOKEN',
    )
    flow.run_config = run_config=DockerRun(
        image='<registry-name>/prefect-runner-base-image:1.0.0',
    )
    When I run this task, I get
    ModuleNotFoundError: No module named 'util'
    . Any help would be much appreciated.
    It seemingly makes sense why this is failing, as
    etl/
    isn't part of the sys.path/pythonpath, so Python doesn't know where to search for the
    util
    module.
    Just wondering if there's an easy way to configure the sys.path/pythonpath to tell python where to look for this module?
    Kevin Kho

    Kevin Kho

    11 months ago
    Hey, you can add it to the PYTHONPATH in the container like this (haven’t done it myself). You can pip install these as a library and they’ll be available for the flow (
    pip install path_to_module
    ). This requires the
    setup.py
    file.
    l

    Lew Dawson

    11 months ago
    @Kevin Kho Thanks for the info. The pythonpath in docker seems interesting. I don't know how prefect materializes the path of github storage. Do you happen to know where prefect mounts the github storage path inside the container?
    Kevin Kho

    Kevin Kho

    11 months ago
    The flow is downloaded from that Storage, and then run as added commands to the container, but I think the path should be dependent on the WORKDIR of the image? I don’t know the path it runs from off the top of my head, but I could check a bit more tomorrow
    l

    Lew Dawson

    11 months ago
    Ahhhh, that's super helpful @Kevin Kho! Hadn't considered the working directory. I'll look into that first. I'll follow up with you to let you know if that resolves it work me, thanks!
    Out of curiosity, where is the code that overlays downloaded Github code into the Docker container? Wanted to take a super quick look. Curious how it's done.
    If you don't know off the top of your head, no worries. You can let me know whenever you are in your IDE next time. Thanks again!
    Kevin Kho

    Kevin Kho

    11 months ago
    You could check this method
    And the
    create_container
    call is here
    l

    Lew Dawson

    11 months ago
    just verified the github storage code does NOT get put into
    WORKDIR
    of container. It must get mounted somewhere else on the image...
    Kevin Kho

    Kevin Kho

    11 months ago
    Ah ok. Thanks for checking and circling back on that!
    l

    Lew Dawson

    11 months ago
    sure. I'll do a little more digging. Let me know what you find tomorrow @Kevin Kho. Thanks much!
    Anna Geller

    Anna Geller

    11 months ago
    @Lew Dawson you could try /root/.prefect/yourFlowName.prefect