Nikhil Jain
06/09/2022, 7:58 PMtask-definition.yaml
I am providing GITHUB_ACCESS_TOKEN
through linux ENVIRONMENT variables. But I am getting this error:
'The secret GITHUB_ACCESS_TOKEN was not found. Please ensure that it was set correctly in your tenant: <https://docs.prefect.io/orchestration/concepts/secrets.html>'`
Am I missing something?Kevin Kho
GITHUB_ACESS_TOKEN
but for prefect to treat it as a secret, it has to be
PREFECT__CONTEXT__SECRETS__GITHUB_ACCESS_TOKEN
? Are you using Local secrets?Nikhil Jain
06/09/2022, 8:50 PMflow.storage = Git(
repo='coveredinc/taskrunner',
flow_path=common.get_flow_path(local_file_path),
repo_host='<http://github.com|github.com>',
git_token_secret_name='GITHUB_ACCESS_TOKEN',
branch_name='git-storage',
add_default_labels=False
)
flow.run = ECSRun(
run_task_kwargs={'cluster': f'{env}-prefect-cluster'},
task_role_arn=f'arn:aws:iam::<>:role/{env}-task-runner-ecs-task-role',
execution_role_arn=f'arn:aws:iam::<>:role/{env}-task-runner-ecs-task-execution-role',
image=f'<>.dkr.ecr.{region}.<http://amazonaws.com/{env}-taskrunner-ecr:latest|amazonaws.com/{env}-taskrunner-ecr:latest>',
task_definition_path=f's3://{env}-prefect-ecs-config/task-definition.yaml',
labels=['ecs', f'{env}']
)
Kevin Kho
Nikhil Jain
06/09/2022, 9:00 PMKevin Kho
Nikhil Jain
06/09/2022, 10:06 PMgit pull
but before the actual flow code is run?Kevin Kho
Nikhil Jain
06/09/2022, 10:09 PMKevin Kho
Nikhil Jain
06/09/2022, 10:14 PMflow.py
depends on another file helper.py
that would not be pulled? So basically the entire code for the flow has to be contained in one file and no imports or dependencies on anything else? What’s the point of Git/Github storage then?Dockerfile
FROM python:3.9
WORKDIR /opt/prefect
COPY requirements.txt setup.py pyproject.toml /opt/prefect/
COPY taskrunner /opt/prefect/taskrunner
ARG GITHUB_ACCESS_TOKEN
RUN git config --global url."https://${GITHUB_ACCESS_TOKEN}:@github.com/".insteadOf "<https://github.com/>" \
&& pip install -r requirements.txt \
&& pip install -e .
Kevin Kho
Nikhil Jain
06/09/2022, 10:21 PMKevin Kho
pip install -e .
, you are adding a folder to the Python path. That clone though is happening in a temporary directory so it’s not in the Python path and not the same directory as the previous installationNikhil Jain
06/09/2022, 10:25 PMKevin Kho
Nikhil Jain
06/09/2022, 10:31 PMCOPY entry.sh /opt/prefect/
ENTRYPOINT [ "./entry.sh" ]
`entry.sh`:
#!/bin/bash
git init
git remote add origin <https://github.com/><your_repo>.git
git fetch
git checkout -t origin/$GITHUB_REPO_BRANCH -f
[[ $REINSTALL_REQUIREMENTS -eq 1 ]] && pip install -r requirements.txt
[[ $REINSTALL_PY_PACKAGE -eq 1 ]] && pip install -e .
exec "$@"
`register_flows.py`: Added GITHUB_REPO_BRANCH
environment variable to the ECS run.
ecs_run = ECSRun(
//... other settings
//...
// Override these env variables to try different code
env={'GITHUB_REPO_BRANCH': 'trunk', 'REINSTALL_REQUIREMENTS': '0', 'REINSTALL_PY_PACKAGE': '0'},
)
Kevin Kho