Mia
03/30/2022, 5:40 PMfrom prefect.storage import GitLab
storage = GitLab(repo="org/repo",
path="/hello/hello_k8s.py",
ref="prefect")
And when I try to run the flow, I get 404 error file not found but the file is there. Is there a way to print out what path prefect runner is using?Kevin Kho
03/30/2022, 5:42 PMAnna Geller
03/30/2022, 5:43 PMKevin Kho
03/30/2022, 5:43 PMMia
03/30/2022, 5:49 PMKevin Kho
03/30/2022, 5:51 PMstorage = GitLab(repo="org/repo",
path="/hello/hello_k8s.py",
ref="prefect")
storage.get_flow()
to see if it works. does it work with something from the main branch if you can test that?Mia
03/30/2022, 5:57 PMKevin Kho
03/30/2022, 6:01 PMMia
03/30/2022, 6:04 PM...envs/prefect_env/lib/python3.8/site-packages/prefect/storage/gitlab.py" line 83, in get_flow
raise ValueError("Flow is not contained in this Storage")
ValueError: Flow is not contained in this Storage
Kevin Kho
03/30/2022, 6:05 PMfrom prefect import Flow, task
import prefect
from prefect.storage.gitlab import GitLab
from prefect.run_configs import KubernetesRun
@task
def abc(x):
return x+1
with Flow("gh storage") as flow:
abc(1)
storage = GitLab(
repo="kvnkho/prefect-gitlab",
path="myflow.py",
ref="main")
storage.add_flow(flow)
storage.get_flow("gh storage")
Mia
03/30/2022, 6:08 PMFailed to load and execute flow run: GitlabGetError('404 Project Not Found')
Anna Geller
03/30/2022, 6:10 PMKevin Kho
03/30/2022, 6:10 PMMia
03/30/2022, 7:22 PMKevin Kho
03/30/2022, 7:28 PMprefect agent kubernetes install
, you can add --env PREFECT___CONTEXT___SECRETS__MYSECRET="VALUE"
and it adds it to the manifest. or you can add it in the job template. I am not confident it gets added with the config.toml
for KubernetesRun. I think it would work though but the env is more sureMia
03/30/2022, 7:47 PMAnna Geller
03/30/2022, 7:49 PMKevin Kho
03/30/2022, 7:49 PMMia
03/30/2022, 10:28 PMKevin Kho
03/30/2022, 10:31 PMMia
03/30/2022, 10:31 PMFailed to load and execute flow run: ValueError('Local Secret "GITLAB-ACCESS-TOKEN" was not found.')
Kevin Kho
03/30/2022, 10:34 PMhost
argument?Mia
03/30/2022, 10:34 PMKevin Kho
03/30/2022, 10:35 PMMia
03/30/2022, 10:43 PM[context.secrets]
GITHUB_ACCESS_TOKEN = "xxx"
didn’t work.
2) I tried agent startup env setup
prefect agent kubernetes install --env PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN="xxx"
didn’t work.
3) tried setting up on my execution environment
export PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN="xxx"
didn’t work.
4) Actually haven’t tried this: Environment variable set in the run config because I didn’t want to save token to the script
5) trie to add it as an environment variable in your Kubernetes job template and also stored as kubernetes secret by
kubectl create secret generic gitlab-access-token --namespace prefect --from-literal=username="mygitlabusername" --from-literal=token="xxx"
and tried to to run kubectl apply -f job_template.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: prefect-server-agent
spec:
template:
spec:
containers:
- name: flow
image: prefecthq/prefect:0.15.12-python3.8
env:
- name: PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN
valueFrom:
secretKeyRef:
name: gitlab-access-token
key: xxx
restartPolicy: Never
which also didn’t work..Kevin Kho
03/30/2022, 11:19 PMMia
03/30/2022, 11:36 PMprefect agent kubernetes start
? When I do that, I get
[2022-03-31 01:32:47+0200] WARNING - prefect.kubernetes | Service host/port is not set. Using out of cluster configuration option.
[2022-03-31 01:32:47+0200] WARNING - prefect.kubernetes | Service host/port is not set. Using out of cluster configuration option.
and then i think it’s just listening for flow runs to be submitted to public prefect cloud. I’d like to run it on my own k8 cluster serverKevin Kho
03/30/2022, 11:54 PMkubectl apply
to spin up the agent. In order to point it to your server, you need the env vars:
PREFECT__BACKEND="server"
PREFECT__SERVER__ENDPOINT=<your-endpoint>
I think you must have done this for the other agent?--job-template
flag along with the install
commandMia
03/31/2022, 12:42 AMprefect agent kubernetes install --env PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN="xxx" PREFECT__BACKEND="server" PREFECT__SERVER__ENDPOINT=<my prefect server endpoint> | kubectl apply --namespace=prefect -f -
above command returns arguments PREFECT__BACKEND and PREFECT__SERVER__ENDPOINT don’t exist
and when i execute it after setting it in the local environment, it creates a prefect-agent pod which keeps crashing.helm install prefect-server prefecthq/prefect-server --namespace prefect -f values.yaml
. This command spawns prefect-server-agent, prefect-server-apollo, prefect-server-create-tenant-job, prefect-server-graphql, prefect-server-hasura, prefect-server-postgresql, prefect-server-towel, and prefect-server-ui. And I’ve updated ~/.prefect/config.toml to point it to my host serverKevin Kho
03/31/2022, 2:34 AM--env
for each env variable you are setting. When you say execute on local environment, do you mean you do:
export PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN="xxx"
and then run the
kubectl apply --namespace=prefect -f -
command after?
What is the error on the crash?
I see on the helm install. You can add the env variable for the secret on the agent in the values.yaml file as well I believe. Although I am personally less experienced with that.Mia
03/31/2022, 11:51 AMTraceback (most recent call last):
File "/usr/local/bin/prefect", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 305, in start
start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)
File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 128, in start_agent
agent.start()
File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 186, in start
self._setup_api_connection()
File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 870, in _setup_api_connection
self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 831, in _register_agent
agent_config_id=self.agent_config_id,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1782, in register_agent
tenant_id=self.tenant_id,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 273, in tenant_id
self._tenant_id = self._get_default_server_tenant()
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 185, in _get_default_server_tenant
response = self.graphql({"query": {"tenant": {"id"}}})
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 458, in graphql
retry_on_api_error=retry_on_api_error,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 414, in post
retry_on_api_error=retry_on_api_error,
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 654, in _request
raise ClientError("Malformed response received from API.") from exc
prefect.exceptions.ClientError: Malformed response received from API.
Anna Geller
03/31/2022, 3:23 PMPREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN
Mia
03/31/2022, 3:32 PMFailed to load and execute flow run: ValueError('Local Secret "GITLAB_ACCESS_TOKEN" was not found.')
Anna Geller
03/31/2022, 3:41 PMexport PREFECT__CLOUD__USE_LOCAL_SECRETS=true
Mia
03/31/2022, 4:02 PMagent:
# enabled determines if the Prefect Kubernetes agent is deployed
enabled: true
# prefectLabels defines what scheduling labels (not K8s labels) should
# be associated with the agent
prefectLabels: []
# jobTemplateFilePath defines which template to use for the agent's jobs. Defaults
# to an empty string, which will use the default template.
# reference: <https://docs.prefect.io/orchestration/agents/kubernetes.html#custom-job-template>
jobTemplateFilePath: ""
# image configures the container image for the agent deployment
image:
name: prefecthq/prefect
tag: null
pullPolicy: Always
pullSecrets: []
labels: {}
annotations: {}
podAnnotations: {}
replicas: 1
strategy: {}
podSecurityContext: {}
securityContext: {}
env: #[]
- name: PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN
value: "xxx"
- name: PREFECT__CLOUD__USE_LOCAL_SECRETS
value: true
nodeSelector: {}
tolerations: []
affinity: {}
after helm install runs successfully, i was able to check that both PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN and PREFECT__CLOUD__USE_LOCAL_SECRETS exists in the agent pod locally as env variables. but the run still fails on this agent with the same local secret not found.Anna Geller
03/31/2022, 4:39 PMKubernetesRun
run configuration in your flow.job_template = """
apiVersion: batch/v1
kind: Job
spec:
template:
spec:
containers:
- name: flow
image: prefecthq/prefect:latest-python3.8
env:
- name: PREFECT__CLOUD__USE_LOCAL_SECRETS
value: true
- name: GITHUB_ACCESS_TOKEN
valueFrom:
secretKeyRef:
name: GITLAB_ACCESS_TOKEN
key: xxx
restartPolicy: Never
"""
and then to apply that on your KubernetesRun:
flow.run_config = KubernetesRun(job_template=job_template)
Mia
04/01/2022, 12:07 AMkey
as GITLAB_ACCESS_TOKEN
and not the actual access token string). That solved the “local secret xyz not found” issue.
Unfortunately, my hello flow is still not working.. onto the next error message(below). I noticed that my agent is trying to create job containers but they are failing. So I checked logs in the agent. :
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 413, in heartbeat
self.manage_jobs()
File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 193, in manage_jobs
f"Job {job.name!r} is for flow run {flow_run_id!r} "
AttributeError: 'V1Job' object has no attribute 'name'
ERROR:agent:Error while managing existing k8s jobs
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 190, in manage_jobs
flow_run_state = self.client.get_flow_run_state(flow_run_id)
File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1340, in get_flow_run_state
raise ObjectNotFoundError(f"Flow run {flow_run_id!r} not found.")
prefect.exceptions.ObjectNotFoundError: Flow run "xxx" not found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 413, in heartbeat
self.manage_jobs()
File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 193, in manage_jobs
f"Job {job.name!r} is for flow run {flow_run_id!r} "
AttributeError: 'V1Job' object has no attribute 'name'
Where could this be coming from?03:51:02
DEBUG
k8s-infra
Event: 'Failed' on pod 'prefect-job-123-123'
Message: Error: couldn't find key gitlab-access-token in Secret prefect/gitlab-access-token
03:51:14
DEBUG
k8s-infra
Event: 'Pulled' on pod 'prefect-job-123-123'
Message: Container image "prefecthq/prefect:latest-python3.8" already present on machine
Kevin Kho
04/01/2022, 2:44 AMflow.storage = GitLab(..., secrets=["GITLAB_ACCESS_TOKEN"]
though I don’t think you should even have to do this, but this will explicitly tell Prefect to load a secret into the contextMia
04/01/2022, 3:22 AMflow.storage = GitLab(..., secrets=["GITLAB_ACCESS_TOKEN"])
, no changes..Kevin Kho
04/01/2022, 3:39 AMMessage: Error: couldn't find key gitlab-access-token in Secret prefect/gitlab-access-token
might be an error from the job template trying to fetch it from the secretKeyRef
- name: GITHUB_ACCESS_TOKEN
valueFrom:
secretKeyRef:
name: GITLAB_ACCESS_TOKEN
key: xxx
Not super sure but the message doesn’t look like a log from the Prefect library. Not exactly sure myself though. Anna would know moreMia
04/01/2022, 10:54 AMjob_template = """
apiVersion: batch/v1
kind: Job
spec:
template:
spec:
containers:
- name: flow
image: prefecthq/prefect:latest-python3.8
env:
- name: PREFECT__CLOUD__USE_LOCAL_SECRETS
value: true
- name: PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN
valueFrom:
secretKeyRef:
name: GITLAB_ACCESS_TOKEN
key: #this should be the key name that you defined in the kubernetes secret
restartPolicy: Never
"""
Anna Geller
04/01/2022, 11:22 AMGITLAB_ACCESS_TOKEN
is the default secret name Prefect is looking for when using this storage. Thanks so much for sharing, and great work solving the issue!Kevin Kho
04/01/2022, 2:06 PM