@Marvin Hello, I am currently trying to use prefect (using helm and values.yaml to setup work-pool and workers). The issue is each flow I try to run crashes due to validation error for KubernetesWorkerJobConfiguration env (value is not a valid dict)

Well, it seems we've hit a bit of a dead end on the GitHub issues front, but the Prefect documentation has provided some insights. The Prefect documentation mentions that KubernetesJob supports various settings, but it doesn't go into detail about the

env

setting. It's possible that the error you're encountering is due to a misconfiguration in your

values.yaml

file. Another part of the documentation provides steps on how to set up a Prefect worker using Helm and a

values.yaml

file. It might be worth revisiting these steps to ensure everything is set up correctly. If you're still encountering issues, it might be beneficial to post a new issue on the Prefect GitHub repository detailing your problem. The Prefect community is quite active and may be able to provide further assistance. And remember, even though computers are supposed to make our lives easier, they sure do have a funny way of making things complicated, don't they?

@Marvin when creating values.yaml to deploy a worker using helm, how can I set the env for that worker?

Ah, the sweet smell of documentation in the morning. Or is it afternoon? Time is such a human construct. Anyway, here's what I found: To set environment variables for a Prefect worker using Helm and a

values.yaml

file, you would typically define them within the

values.yaml

file itself. However, the Prefect documentation doesn't provide explicit instructions on how to do this. The documentation does provide steps on how to set up a Prefect worker using Helm and a

values.yaml

file, but it doesn't go into detail about setting environment variables specifically. Here's a general example of how you might define environment variables in a

values.yaml

file:

yaml
env:
  - name: ENV_VAR_NAME
    value: "env_var_value"

You would replace

ENV_VAR_NAME

with the name of your environment variable and

env_var_value

with its value. Please note that this is a general example and might not work for your specific use case. If you're still having trouble, I recommend reaching out to the Prefect community for more specific guidance. And remember, YAML might seem like a maze, but at least it's not XML.

@Marvin I am getting 403 error when prefect tries to create the job. How can I set the default namespace for the jobs or allow creation from k8s? this is the error: kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'ce2ba96e-3412-4e57-be0a-411501401b01', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '8bb77861-d3b0-419e-a3d8-737a7de19f18', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e35deae4-4844-45e4-8fa1-a7fb9be50be9', 'Date': 'Wed, 12 Jul 2023 111613 GMT', 'Content-Length': '318'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"systemserviceaccountprefect:prefect-worker\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}

Ah, the forbidden fruit of Kubernetes. It seems you're not alone in this garden of confusion. The error message you're seeing suggests that the Prefect worker doesn't have the necessary permissions to create jobs in the default namespace. This is likely due to Kubernetes Role-Based Access Control (RBAC) settings. According to a GitHub issue, you can resolve this by applying the necessary RBAC to your namespace. You can use the

prefect agent install kubernetes -t TOKEN --namespace foo --rbac

command to automatically apply the RBAC when installing the agent. Replace

TOKEN

with your Prefect API token and

foo

with your namespace. If you've already installed the agent, you can manually apply the RBAC. Copy the RBAC from the Prefect documentation, update the namespace, and put it into a YAML file. Then, apply it using

kubectl apply -f

. Remember to specify the namespace when applying the agent with

kubectl

. The

prefect agent install --namespace

command tells the agent where to create jobs, while the

kubectl apply --namespace

command tells Kubernetes where to put the agent. For example:

prefect agent install kubernetes ... | kubectl apply --namespace=prefect -f -

. And remember, even in the world of Kubernetes, it's not always the user who is in error, but sometimes the system. Or is it always the system?

The chart should be deployed to the namespace you deployed it to at installation time; the service account created only gets permissions to that namespace

thank you so much, worked like a charm ^^

Well, it seems the Prefect documentation is playing a bit hard to get. It doesn't provide a clear example of how to reference a GitLabRepository block or GitlabCredentials block in the

prefect.deployments.steps.git_clone

step. However, based on the general structure of Prefect, you would typically reference a block in the following format:

{{ blocks.block-type/block-name }}

. So, if you have a GitLabRepository block named

my-repo

, you would reference it as

{{ blocks.GitLabRepository/my-repo }}

. As for the GitlabCredentials block, it's not clear from the documentation how this would be referenced. I would assume it follows a similar pattern, but without concrete examples, it's hard to say for sure. If you're still having trouble, I recommend reaching out to the Prefect community for more specific guidance. They might be able to provide more detailed examples or point you to resources that I wasn't able to find. And remember, even in the world of software development, sometimes the answers aren't as straightforward as we'd like them to be. It's all part of the fun, right?