I am trying to use GIT storage with ECS run. The goal is to have a docker container which has all the python packages installed and then just pull the code from github prior to running the flow. This obviously requires that the container should be able to pull code from Github. Then, in

task-definition.yaml

I am providing

GITHUB_ACCESS_TOKEN

through linux ENVIRONMENT variables. But I am getting this error:

'The secret GITHUB_ACCESS_TOKEN was not found.  Please ensure that it was set correctly in your tenant: <https://docs.prefect.io/orchestration/concepts/secrets.html>'`

Am I missing something?

I think you are providing

GITHUB_ACESS_TOKEN

but for prefect to treat it as a secret, it has to be

PREFECT__CONTEXT__SECRETS__GITHUB_ACCESS_TOKEN

? Are you using Local secrets?

I have this setup:

flow.storage = Git(
        repo='coveredinc/taskrunner',
        flow_path=common.get_flow_path(local_file_path),
        repo_host='<http://github.com|github.com>',
        git_token_secret_name='GITHUB_ACCESS_TOKEN',
        branch_name='git-storage',
        add_default_labels=False
    )

flow.run = ECSRun(
    run_task_kwargs={'cluster': f'{env}-prefect-cluster'},
    task_role_arn=f'arn:aws:iam::<>:role/{env}-task-runner-ecs-task-role',
    execution_role_arn=f'arn:aws:iam::<>:role/{env}-task-runner-ecs-task-execution-role',
    image=f'<>.dkr.ecr.{region}.<http://amazonaws.com/{env}-taskrunner-ecr:latest|amazonaws.com/{env}-taskrunner-ecr:latest>',
    task_definition_path=f's3://{env}-prefect-ecs-config/task-definition.yaml',
    labels=['ecs', f'{env}']
)

Yes but Local Secrets or Secrets hosted with Prefect Cloud?

Sorry, that edit was just to cleanup the main thread. I haven’t setup any secrets in Prefect Cloud so I am guessing it’s a local secret? I was trying out the suggestion you had, it seems git pull worked out fine. Now my next challenge will be to get the code to work (especially fixing “ModuleNotFound” errors).

ModuleNotFound is likely you don’t have packages in the image right?

@Kevin Kho Actually I am not getting ModuleNotFound error. But any updates to the code are not getting reflected in flow run. I think that’s because the code is being “packaged” into a python package. Even if I install it in editable mode, code updates pulled via git are not applied to the package. Any thoughts on getting around this? Is there a way to run some kind of ‘setup’ code after

git pull

but before the actual flow code is run?

I think you saw my article about packaging a module into Docker right? Taht’s what you tried?

yes

Can you share the Dockerfile?

hmm… so if my

flow.py

depends on another file

helper.py

that would not be pulled? So basically the entire code for the flow has to be contained in one file and no imports or dependencies on anything else? What’s the point of Git/Github storage then?

Not pulled by Git storage. It needs to be in the Docker image. The point of those is just to pull the Flow file. Prefect does not install a Git repo into a module, we leave that up to the user because we’d need to make a lot of assumptions and we’d be recreating Python packaging to achieve that (for 1.0). For 2.0, we are looking into simplifying that packaging story. You’d have to manipulate the Python path to get things as a module, but it’s very hard (might be impossible to do it)

I don’t understand, do you inspect the filenames in the Git repo and selectively pull only non-python files? I’d expect prefect to simply do a git pull on the branch name provided. As for modules, I was hoping that if I install the package in “editable” mode while creating the docker image, then any changes “pulled” from the repo should be immediately reflected without having to install the package again. btw, I just realized there is an issue with my directory structure. i.e. git repo structure is not the same as dir structure in docker image.

No we clone it. But can’t install it. So when you do

pip install -e .

, you are adding a folder to the Python path. That clone though is happening in a temporary directory so it’s not in the Python path and not the same directory as the previous installation

aah… I see. It’s being cloned into a temp dir. hmm…

What I have seen someone do is add the clone and install as the ENTRYPOINT for the Docker container. You can try that?

hmm… worth a try.

Ah nice work!