I’m running into an issue trying to use GitLab storage to reference flows stored in my gitlab repo. This is how I have it in my script:

from prefect.storage import GitLab
storage = GitLab(repo="org/repo",
                 path="/hello/hello_k8s.py",
                 ref="prefect")

And when I try to run the flow, I get 404 error file not found but the file is there. Is there a way to print out what path prefect runner is using?

The call is pretty easy to replicate if you check the code here. But I have an example one sec.

you also have a typo - is it a copy-paste issue? it should be GitLab not Gitlab

I believe this works. Can you try removing your first slash in the path?

@Kevin Kho I tried removing the first slash in the path but still doesn’t work. @Anna Geller sorry it was a typo here. I had it GitLab.

So to test this easily, you can do:

storage = GitLab(repo="org/repo",
                 path="/hello/hello_k8s.py",
                 ref="prefect")

storage.get_flow()

to see if it works. does it work with something from the main branch if you can test that?

do I also have to add flow name to storage.get_flow()?

Ah yes

I’m getting following error when i try to do storage.get_flow(<flowname>)

...envs/prefect_env/lib/python3.8/site-packages/prefect/storage/gitlab.py" line 83, in get_flow
    raise ValueError("Flow is not contained in this Storage")
  ValueError: Flow is not contained in this Storage

Here is a working example:

from prefect import Flow, task
import prefect
from prefect.storage.gitlab import GitLab
from prefect.run_configs import KubernetesRun

@task
def abc(x):
    return x+1

with Flow("gh storage") as flow:
    abc(1)

storage = GitLab(
repo="kvnkho/prefect-gitlab", 
path="myflow.py",
ref="main")

storage.add_flow(flow)
storage.get_flow("gh storage")

no i get this

Failed to load and execute flow run: GitlabGetError('404 Project Not Found')

check this https://discourse.prefect.io/t/i-got-an-error-failed-to-load-and-execute-flows-environment-gitlabgeterror-404-project-not-found-how-to-fix-it/112

If it’s project, it can’t even find the repo. If it’s commit, it can’t find the branch

For the kubernetes deployment, do I configure the secret in config.toml?

better if on the agent. when you do

prefect agent kubernetes install

, you can add

--env PREFECT___CONTEXT___SECRETS__MYSECRET="VALUE"

and it adds it to the manifest. or you can add it in the job template. I am not confident it gets added with the

config.toml

for KubernetesRun. I think it would work though but the env is more sure

this would be the case when i’m running the server?? What if I have the k8s remotely set up with Helm? do I store it as k8s secrets in the namespace? or do I update values.yaml file and upgrade?

here are several ways you can do it - using an environment variable is the main way of approaching this https://discourse.prefect.io/t/how-to-set-secrets-e-g-github-access-token-on-server/70

Yeah everything in the config.toml can be specified with the env variable. For Server, you only have local secrets so you can define it with the env variable on the agent like mentioned here . And then the agent will pass this env variable to the Flow run.

I’ve tried all the five steps in Anna’s link but they all didn’t work out (the setup i have is with K8s with Helm Chart). I’ve also tried out creating secret with kubectl and tried to pass it from there but that also didn’t work. I’m not sure how else to check😥

is that a private GitLab or public?

private

Oh this hits the public one though. Did you specify the

host

argument?

yes i did

Ah ok how did you specify the secret?

i tried specifying in 1) config.toml like below

[context.secrets]
GITHUB_ACCESS_TOKEN = "xxx"

didn’t work. 2) I tried agent startup env setup

prefect agent kubernetes install --env PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN="xxx"

didn’t work. 3) tried setting up on my execution environment

export PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN="xxx"

didn’t work. 4) Actually haven’t tried this: Environment variable set in the run config because I didn’t want to save token to the script 5) trie to add it as an environment variable in your Kubernetes job template and also stored as kubernetes secret by

kubectl create secret generic gitlab-access-token --namespace prefect --from-literal=username="mygitlabusername" --from-literal=token="xxx"

and tried to to run

kubectl apply -f job_template.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: prefect-server-agent
spec:
  template:
    spec:
      containers:
        - name: flow
          image: prefecthq/prefect:0.15.12-python3.8
          env:
            - name: PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN
              valueFrom:
                secretKeyRef:
                  name: gitlab-access-token
                  key: xxx
      restartPolicy: Never

which also didn’t work..

You re-redeployed the agent after number 2? 3 will not work because the context is made at the start of the flow so putting it in the execution environment is too late

when you say re-deplyed, do you mean

prefect agent kubernetes start

? When I do that, I get

[2022-03-31 01:32:47+0200] WARNING - prefect.kubernetes | Service host/port is not set. Using out of cluster configuration option.
[2022-03-31 01:32:47+0200] WARNING - prefect.kubernetes | Service host/port is not set. Using out of cluster configuration option.

and then i think it’s just listening for flow runs to be submitted to public prefect cloud. I’d like to run it on my own k8 cluster server

How did you get the previous agent? So the step in number 2 just creates the manifest but you still have to apply it to your cluster. The instructions are here . So you need to make a new agent that will have the secret and then when the Flows get picked up by the new agent, the will get the local secret. So you do the install command and then pass it to

kubectl apply

to spin up the agent. In order to point it to your server, you need the env vars:

PREFECT__BACKEND="server"
PREFECT__SERVER__ENDPOINT=<your-endpoint>

I think you must have done this for the other agent?

Do you mean as a local environment? or pass it as environment variable to prefect agent install?

prefect agent kubernetes install --env PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN="xxx" PREFECT__BACKEND="server" PREFECT__SERVER__ENDPOINT=<my prefect server endpoint> | kubectl apply --namespace=prefect -f -

above command returns arguments PREFECT__BACKEND and PREFECT__SERVER__ENDPOINT don’t exist and when i execute it after setting it in the local environment, it creates a prefect-agent pod which keeps crashing.

Yes like the command you mentioned. You need to do

--env

for each env variable you are setting. When you say execute on local environment, do you mean you do:

export PREFECT__CONTEXT__SECRETS__GITLAB-ACCESS-TOKEN="xxx"

and then run the

kubectl apply --namespace=prefect -f -

command after? What is the error on the crash? I see on the helm install. You can add the env variable for the secret on the agent in the values.yaml file as well I believe. Although I am personally less experienced with that.

This is the logs i get from pods crash

Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 305, in start
    start_agent(KubernetesAgent, image_pull_secrets=image_pull_secrets, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/agent.py", line 128, in start_agent
    agent.start()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 186, in start
    self._setup_api_connection()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 870, in _setup_api_connection
    self.client.attach_headers({"X-PREFECT-AGENT-ID": self._register_agent()})
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 831, in _register_agent
    agent_config_id=self.agent_config_id,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1782, in register_agent
    tenant_id=self.tenant_id,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 273, in tenant_id
    self._tenant_id = self._get_default_server_tenant()
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 185, in _get_default_server_tenant
    response = self.graphql({"query": {"tenant": {"id"}}})
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 458, in graphql
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 414, in post
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 654, in _request
    raise ClientError("Malformed response received from API.") from exc
prefect.exceptions.ClientError: Malformed response received from API.

it should be underscores:

PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN

I’ve tried both

how exactly did you set it on your helm chart?

This is a snippet of my value.yaml for helm install/upgrade

agent:
  # enabled determines if the Prefect Kubernetes agent is deployed
  enabled: true

  # prefectLabels defines what scheduling labels (not K8s labels) should
  # be associated with the agent
  prefectLabels: []

  # jobTemplateFilePath defines which template to use for the agent's jobs. Defaults
  # to an empty string, which will use the default template.
  # reference: <https://docs.prefect.io/orchestration/agents/kubernetes.html#custom-job-template>
  jobTemplateFilePath: ""

  # image configures the container image for the agent deployment
  image:
    name: prefecthq/prefect
    tag: null
    pullPolicy: Always
    pullSecrets: []
   

  labels: {}
  annotations: {}
  podAnnotations: {}
  replicas: 1
  strategy: {}
  podSecurityContext: {}
  securityContext: {}
  env: #[]
    - name: PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN 
      value: "xxx"
    - name: PREFECT__CLOUD__USE_LOCAL_SECRETS
      value: true
  nodeSelector: {}
  tolerations: []
  affinity: {}

after helm install runs successfully, i was able to check that both PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN and PREFECT__CLOUD__USE_LOCAL_SECRETS exists in the agent pod locally as env variables. but the run still fails on this agent with the same local secret not found.

Nice work checking that and ensuring that the agent has that environment variable. It looks like it works well, but it doesn't propagate to the Kubernetes flow run pod. To fix this, you could add a Kubernetes job template to your

KubernetesRun

run configuration in your flow.

That worked for now. Thank you @Kevin Khoand @Anna Geller. So to summarize, I’ve updated my values.yaml for helm chart in the agent section to pass env variables, and then updated my hello_k8s.py script with job_template variable defined (the only thing i changed in Anna’s example above was setting

key

as

GITLAB_ACCESS_TOKEN

and not the actual access token string). That solved the “local secret xyz not found” issue. Unfortunately, my hello flow is still not working.. onto the next error message(below). I noticed that my agent is trying to create job containers but they are failing. So I checked logs in the agent. :

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 413, in heartbeat
    self.manage_jobs()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 193, in manage_jobs
    f"Job {job.name!r} is for flow run {flow_run_id!r} "
AttributeError: 'V1Job' object has no attribute 'name'
ERROR:agent:Error while managing existing k8s jobs
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 190, in manage_jobs
    flow_run_state = self.client.get_flow_run_state(flow_run_id)
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1340, in get_flow_run_state
    raise ObjectNotFoundError(f"Flow run {flow_run_id!r} not found.")
prefect.exceptions.ObjectNotFoundError: Flow run "xxx" not found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 413, in heartbeat
    self.manage_jobs()
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 193, in manage_jobs
    f"Job {job.name!r} is for flow run {flow_run_id!r} "
AttributeError: 'V1Job' object has no attribute 'name'

Where could this be coming from?

Seems like there is an open issue for the AtteibuteError you have. I think this might be because of the latest version of kubernetes. Maybe you can try downgrading your library a version?

isn’t the error coming from prefect? I’ve also tried adding

flow.storage = GitLab(..., secrets=["GITLAB_ACCESS_TOKEN"])

, no changes..

The first traceback with the attribute error is more like Prefect calls a method/attribute that doesnt exist in a newer version of the package. I am personally pretty confused since it seems like we went through all the options. The RunConfig should be super sure already of it getting into the pod. I looked at this again, and I think

Message: Error: couldn't find key gitlab-access-token in Secret prefect/gitlab-access-token

might be an error from the job template trying to fetch it from the secretKeyRef

- name: GITHUB_ACCESS_TOKEN
            valueFrom:
              secretKeyRef:
                name: GITLAB_ACCESS_TOKEN
                key: xxx

Not super sure but the message doesn’t look like a log from the Prefect library. Not exactly sure myself though. Anna would know more

That took me a while! I finally got the hello example to work. I didn’t need to pass “secret=” argument in the GitLab() call in my prefect script. This time it was just getting the right key names in the job_template

job_template = """
apiVersion: batch/v1
kind: Job
spec:
  template:
    spec:
      containers:
      - name: flow
        image: prefecthq/prefect:latest-python3.8
        env:
          - name: PREFECT__CLOUD__USE_LOCAL_SECRETS
            value: true
          - name: PREFECT__CONTEXT__SECRETS__GITLAB_ACCESS_TOKEN
            valueFrom:
              secretKeyRef:
                name: GITLAB_ACCESS_TOKEN
                key: #this should be the key name that you defined in the kubernetes secret
      restartPolicy: Never
"""

This makes so much sense since

GITLAB_ACCESS_TOKEN

is the default secret name Prefect is looking for when using this storage. Thanks so much for sharing, and great work solving the issue!

Ah I see. Thanks for sharing!