Afternoon folks. Wondering if anyone has advice. Struggling to get a simple/common Python pattern working when using

DockerRun

run_config with

Github

storage. Have the following project structure in the Github repo:

etl/
├── util/
│   ├── __init__.py
│   ├── util_vault.py
├── __init__.py
├── runner.py

The

util_vault.py

looks like:

from prefect.client import Secret

class UtilVault:

    @staticmethod
    def get_secret_simple(name):
        return Secret(name).get()

The

runner.py

looks like:

from prefect import Flow, task, context
from prefect.run_configs import DockerRun
from prefect.storage import GitHub

from util.util_vault import UtilVault

logger = context.get("logger")


@task
def hello_secrets():
    secret = UtilVault.get_secret_simple('GITHUB_ACCESS_TOKEN')
    <http://logger.info|logger.info>('secret = %s', secret)


with Flow(name='hello_world') as flow:
    hello_secrets()

flow.storage = GitHub(
    repo='<username>/<repo>',
    path='etl/runner.py',
    access_token_secret='GITHUB_ACCESS_TOKEN',
)
flow.run_config = run_config=DockerRun(
    image='<registry-name>/prefect-runner-base-image:1.0.0',
)

When I run this task, I get

ModuleNotFoundError: No module named 'util'

. Any help would be much appreciated.

Hey, you can add it to the PYTHONPATH in the container like this (haven’t done it myself). You can pip install these as a library and they’ll be available for the flow (

pip install path_to_module

). This requires the

setup.py

file.

@Kevin Kho Thanks for the info. The pythonpath in docker seems interesting. I don't know how prefect materializes the path of github storage. Do you happen to know where prefect mounts the github storage path inside the container?

The flow is downloaded from that Storage, and then run as added commands to the container, but I think the path should be dependent on the WORKDIR of the image? I don’t know the path it runs from off the top of my head, but I could check a bit more tomorrow

Ahhhh, that's super helpful @Kevin Kho! Hadn't considered the working directory. I'll look into that first. I'll follow up with you to let you know if that resolves it work me, thanks!

You could check this method

just verified the github storage code does NOT get put into

WORKDIR

of container. It must get mounted somewhere else on the image...

Ah ok. Thanks for checking and circling back on that!

sure. I'll do a little more digging. Let me know what you find tomorrow @Kevin Kho. Thanks much!

@Lew Dawson you could try /root/.prefect/yourFlowName.prefect