https://prefect.io logo
k

Karim Dahmeni

07/12/2023, 10:19 AM
@Marvin Hello, I am currently trying to use prefect (using helm and values.yaml to setup work-pool and workers). The issue is each flow I try to run crashes due to validation error for KubernetesWorkerJobConfiguration env (value is not a valid dict)
m

Marvin

07/12/2023, 10:20 AM
Well, it seems we've hit a bit of a dead end on the GitHub issues front, but the Prefect documentation has provided some insights. The Prefect documentation mentions that KubernetesJob supports various settings, but it doesn't go into detail about the
env
setting. It's possible that the error you're encountering is due to a misconfiguration in your
values.yaml
file. Another part of the documentation provides steps on how to set up a Prefect worker using Helm and a
values.yaml
file. It might be worth revisiting these steps to ensure everything is set up correctly. If you're still encountering issues, it might be beneficial to post a new issue on the Prefect GitHub repository detailing your problem. The Prefect community is quite active and may be able to provide further assistance. And remember, even though computers are supposed to make our lives easier, they sure do have a funny way of making things complicated, don't they?
k

Karim Dahmeni

07/12/2023, 10:50 AM
@Marvin when creating values.yaml to deploy a worker using helm, how can I set the env for that worker?
m

Marvin

07/12/2023, 10:51 AM
Ah, the sweet smell of documentation in the morning. Or is it afternoon? Time is such a human construct. Anyway, here's what I found: To set environment variables for a Prefect worker using Helm and a
values.yaml
file, you would typically define them within the
values.yaml
file itself. However, the Prefect documentation doesn't provide explicit instructions on how to do this. The documentation does provide steps on how to set up a Prefect worker using Helm and a
values.yaml
file, but it doesn't go into detail about setting environment variables specifically. Here's a general example of how you might define environment variables in a
values.yaml
file:
Copy code
yaml
env:
  - name: ENV_VAR_NAME
    value: "env_var_value"
You would replace
ENV_VAR_NAME
with the name of your environment variable and
env_var_value
with its value. Please note that this is a general example and might not work for your specific use case. If you're still having trouble, I recommend reaching out to the Prefect community for more specific guidance. And remember, YAML might seem like a maze, but at least it's not XML.
k

Karim Dahmeni

07/12/2023, 11:18 AM
@Marvin I am getting 403 error when prefect tries to create the job. How can I set the default namespace for the jobs or allow creation from k8s? this is the error: kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'ce2ba96e-3412-4e57-be0a-411501401b01', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '8bb77861-d3b0-419e-a3d8-737a7de19f18', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e35deae4-4844-45e4-8fa1-a7fb9be50be9', 'Date': 'Wed, 12 Jul 2023 111613 GMT', 'Content-Length': '318'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"systemserviceaccountprefect:prefect-worker\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
m

Marvin

07/12/2023, 11:19 AM
Ah, the forbidden fruit of Kubernetes. It seems you're not alone in this garden of confusion. The error message you're seeing suggests that the Prefect worker doesn't have the necessary permissions to create jobs in the default namespace. This is likely due to Kubernetes Role-Based Access Control (RBAC) settings. According to a GitHub issue, you can resolve this by applying the necessary RBAC to your namespace. You can use the
prefect agent install kubernetes -t TOKEN --namespace foo --rbac
command to automatically apply the RBAC when installing the agent. Replace
TOKEN
with your Prefect API token and
foo
with your namespace. If you've already installed the agent, you can manually apply the RBAC. Copy the RBAC from the Prefect documentation, update the namespace, and put it into a YAML file. Then, apply it using
kubectl apply -f
. Remember to specify the namespace when applying the agent with
kubectl
. The
prefect agent install --namespace
command tells the agent where to create jobs, while the
kubectl apply --namespace
command tells Kubernetes where to put the agent. For example:
prefect agent install kubernetes ... | kubectl apply --namespace=prefect -f -
. And remember, even in the world of Kubernetes, it's not always the user who is in error, but sometimes the system. Or is it always the system?
c

Christopher Boyd

07/12/2023, 11:53 AM
The chart should be deployed to the namespace you deployed it to at installation time; the service account created only gets permissions to that namespace
You should be able to specify in the work-pool (in the prefect UI) both: A) the service account to use B) the namespace to use
k

Karim Dahmeni

07/12/2023, 12:31 PM
thank you so much, worked like a charm ^^
🙌 1
@Marvin in prefect.deployments.steps.git_clone step, what should I write to reference a GitLabRepository block or GitlabCredentials block?
m

Marvin

07/12/2023, 1:47 PM
Well, it seems the Prefect documentation is playing a bit hard to get. It doesn't provide a clear example of how to reference a GitLabRepository block or GitlabCredentials block in the
prefect.deployments.steps.git_clone
step. However, based on the general structure of Prefect, you would typically reference a block in the following format:
{{ blocks.block-type/block-name }}
. So, if you have a GitLabRepository block named
my-repo
, you would reference it as
{{ blocks.GitLabRepository/my-repo }}
. As for the GitlabCredentials block, it's not clear from the documentation how this would be referenced. I would assume it follows a similar pattern, but without concrete examples, it's hard to say for sure. If you're still having trouble, I recommend reaching out to the Prefect community for more specific guidance. They might be able to provide more detailed examples or point you to resources that I wasn't able to find. And remember, even in the world of software development, sometimes the answers aren't as straightforward as we'd like them to be. It's all part of the fun, right?
6 Views