Baris Cekic
04/27/2022, 12:30 PMKubernetesRun
. The flow is trying to download the py
file from a local git repo in the same local K8S cluster. But I got an SSL_VERIFICATION
error even though the url/host
for git storage
is HTTP
. Is there anyone having similar issue?Anna Geller
Baris Cekic
04/27/2022, 12:54 PMflow.storage = Git(repo="flows", flow_path="prefect_flow_1.py", repo_host="repo.default.svc.cluster.local")
dulwich.porcelain
Kevin Kho
Baris Cekic
04/27/2022, 2:53 PMflow.storage = Git(repo="flows", flow_path="prefect_flow_1.py", repo_host="repo.default.svc.cluster.local:3000")
Error message
Failed to load and execute flow run: MaxRetryError("HTTPSConnectionPool(host='repo.default.svc.cluster.local', port=3000): Max retries exceeded with url: /flows.git/info/refs?service=git-upload-pack (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1091)')))")
<http://repo.default.svc.cluster.local:3000>
but it always tries to use HTTPS regardless of selection here.dulwich.porcelain
uses requests
without verify=True|False
switch, at least what I see in prefect repo.Kevin Kho
Baris Cekic
04/27/2022, 3:52 PMNote that this configuration only affects the environment in which it's configured. So if you set values locally, they'll affect flows run locally or via a local agent, but not flows deployed via other agents (since those flow runs happen in a different environment). To set local secrets on flow runs deployed by an agent, you can use the --env flag to forward environment variables into the flow run environment.
Should I maintain environment variables of agent instead of ~/.prefect/config.toml
?Kevin Kho
prefect agent kubernetes install
which makes the template, but you can also do
prefect agent kubernetes install --env PREFECT__CONTEXT__SECRETS__MYSECRET="mysecret"
and then have it populate in the template. Basically it’s likely easier to use the env var to create the secret like shown here
If you need more than 1 secret, just repeat another --env
callstorage = Git(git_clone_url_secret_name="MYSECRET")
and then it will pull that secret to clone the repo during run timeBaris Cekic
04/27/2022, 3:59 PMKevin Kho
Baris Cekic
04/27/2022, 3:59 PMKevin Kho
Baris Cekic
04/27/2022, 3:59 PMIMAGE_PULL_POLICY:
IMAGE_PULL_SECRETS:
JOB_CPU_LIMIT:
JOB_CPU_REQUEST:
JOB_MEM_LIMIT:
JOB_MEM_REQUEST:
NAMESPACE: sandbox
PREFECT__BACKEND: server
PREFECT__CLOUD__AGENT__AGENT_ADDRESS: <http://0.0.0.0:8080>
PREFECT__CLOUD__AGENT__LABELS: ["prefect-k8s"]
PREFECT__CLOUD__API: <http://prefect-server-apollo.sandbox:4200/graphql>
PREFECT__CONTEXT__SECRETS__REPO_URL: <http://repo.default.svc.cluster.local:3000>
SERVICE_ACCOUNT_NAME: prefect-server-serviceaccount
but somehow getting this error
Failed to load and execute flow run: ValueError('Local Secret "REPO_URL" was not found.')
Kevin Kho
Baris Cekic
04/27/2022, 5:12 PMKevin Kho
flow.storage = Git(..., secrets=["REPO_URL"])
This will tell context to include that secretBaris Cekic
04/27/2022, 5:12 PMflow.storage = Git(repo="flows", flow_path="prefect_flow_1.py", git_clone_url_secret_name="REPO_URL")
git_clone_url_secret_name
with secrets=["REPO_URL"]
Kevin Kho
secrets
is another kwargBaris Cekic
04/27/2022, 5:14 PMKevin Kho
Baris Cekic
04/27/2022, 5:15 PM18:14:58
lens
WARNING
Git
Git storage initialized with a `git_clone_url_secret_name`. The value of this Secret will be used to clone the repository, ignoring `repo`, `repo_host`, `git_token_secret_name`, `git_token_username`, `use_ssh`, and `format_access_token`.
18:14:58
lens
ERROR
execute flow-run
Failed to load and execute flow run: ValueError('Local Secret "REPO_URL" was not found.')
Kevin Kho
Baris Cekic
04/27/2022, 7:13 PMPREFECT__BACKEND: server
PREFECT__CLOUD__AGENT__LABELS: ['prefect-k8s']
PREFECT__CLOUD__API: <http://prefect-server-apollo.sandbox:4200/graphql>
PREFECT__CLOUD__API_KEY:
PREFECT__CLOUD__AUTH_TOKEN:
PREFECT__CLOUD__SEND_FLOW_RUN_LOGS: true
PREFECT__CLOUD__TENANT_ID:
PREFECT__CLOUD__USE_LOCAL_SECRETS: false
PREFECT__CONTEXT__FLOW_ID: 9cf35e89-c1df-4fd6-841a-10650784c311
PREFECT__CONTEXT__FLOW_RUN_ID: be8b3e7b-e985-4e6b-abcc-deadfcef39f3
PREFECT__CONTEXT__IMAGE: prefecthq/prefect:1.2.0
PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS: prefect.engine.cloud.CloudFlowRunner
PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS: prefect.engine.cloud.CloudTaskRunner
PREFECT__LOGGING__LEVEL: INFO
PREFECT__LOGGING__LOG_TO_CLOUD: true
PREFECT__CONTEXT__SECRETS__REPO_URL: <http://repo.default.svc.cluster.local:3000>
Kevin Kho
Baris Cekic
04/27/2022, 7:21 PMprefecthq/prefect:latest
run_config
of KubernetesRun()
Kevin Kho
run_config
of KubernetesRun
will absolutely work, just the least secure of courseBaris Cekic
04/27/2022, 7:29 PMIMAGE_PULL_POLICY:
IMAGE_PULL_SECRETS:
JOB_CPU_LIMIT:
JOB_CPU_REQUEST:
JOB_MEM_LIMIT:
JOB_MEM_REQUEST:
NAMESPACE: sandbox
PREFECT__BACKEND: server
PREFECT__CLOUD__AGENT__AGENT_ADDRESS: <http://0.0.0.0:8080>
PREFECT__CLOUD__AGENT__LABELS: ["prefect-k8s"]
PREFECT__CLOUD__API: <http://prefect-server-apollo.sandbox:4200/graphql>
PREFECT__CONTEXT__SECRETS__REPO_URL: <http://repo.default.svc.cluster.local:3000>
SERVICE_ACCOUNT_NAME: prefect-server-serviceaccount
Kevin Kho
Baris Cekic
04/27/2022, 7:31 PMKevin Kho
Baris Cekic
04/27/2022, 7:33 PM$ env
with Flow("basic-prefect-etl-flow", run_config=KubernetesRun(labels=["prefect-k8s"]),
#storage=GitHub(repo="cekicbaris/public", path="prefect_flow_1.py")
#storage=Docker(python_dependencies=["pandas==1.1.0"],image_tag='latest')
) as flow:
extracted_df = extract()
transformed_df = transform(extracted_df)
load(transformed_df)
flow.storage = Git(repo="flows", flow_path="prefect_flow_1.py",
, git_clone_url_secret_name="REPOURL", secrets=["REPOURL"])
if __name__ == '__main__':
# flow.run()
flow.register(project_name='Test')
INFO:agent:Deploying flow run be8b3e7b-e985-4e6b-abcc-deadfcef39f3 to execution environment...
WARNING:prefect.Git:Git storage initialized with a `git_clone_url_secret_name`. The value of this Secret will be used to clone the repository, ignoring `repo`, `repo_host`, `git_token_secret_name`, `git_token_username`, `use_ssh`, and `format_access_token`.
[2022-04-27 19:11:01+0000] WARNING - prefect.Git | Git storage initialized with a `git_clone_url_secret_name`. The value of this Secret will be used to clone the repository, ignoring `repo`, `repo_host`, `git_token_secret_name`, `git_token_username`, `use_ssh`, and `format_access_token`.
[2022-04-27 19:11:02,123] INFO - agent | Completed deployment of flow run be8b3e7b-e985-4e6b-abcc-deadfcef39f3
INFO:agent:Completed deployment of flow run be8b3e7b-e985-4e6b-abcc-deadfcef39f3
Kevin Kho
REPO_URL
instead of REPOURL
?Baris Cekic
04/27/2022, 7:36 PMREPOURL
or REPO_URL
UI logs
here is with new secret,
Failed to load and execute flow run: ValueError('Local Secret "REPOURL" was not found.')
here is with old secret
Failed to load and execute flow run: ValueError('Local Secret "REPO_URL" was not found.')
Kevin Kho
Baris Cekic
04/27/2022, 7:39 PMKevin Kho
Baris Cekic
04/27/2022, 9:26 PMrun_config
Kevin Kho
Baris Cekic
04/27/2022, 9:44 PMPREFECT__CLOUD__AGENT__ENV_VARS
and I made the change as below.
env:
- name: PREFECT__CLOUD__AGENT__ENV_VARS
value: '{"PREFECT__CONTEXT__SECRETS__REPO_URL": "<http://repo.default.svc.cluster.local:3000/repo1/flows>"}'
I figured out the difference when I run the kubernetes agent install
prefect agent kubernetes install --env PREFECT__CONTEXT__SECRETS__REPO_URL="<http://repo.default.svc.cluster.local:3000/repo1/flows>"
Then when it creates the kubernetes job , it moves the secret to the job.
🎉Kevin Kho