https://prefect.io logo
Powered by
Title
b

Brett

04/21/2023, 5:45 PM
Good day! I'm trying to convert my Prefect 1 to Prefect 2. In 1 I was able to run an agent pod in kubernetes and just assign it a job template yaml file to use to run flows. I'd like to do something similar in 2. I've read a little bit about infrastructure blocks, but ideally I just want to define it with the same job yaml file. Is this possible to do in Prefect 2 without an infrastructure block?
✅ 1
k

Kevin Grismore

04/21/2023, 6:20 PM
rather than an agent, you can use a kubernetes work pool with an attached worker that has a base job definition instead of creating infra blocks for all your deployments
🙌 1
https://prefecthq.github.io/prefect-kubernetes/worker/
https://docs.prefect.io/latest/concepts/work-pools/
b

Brett

04/21/2023, 6:37 PM
I've been playing around with that too. So I'd define a worker. Then would be able to give it a job template yaml that includes an image I want it to use and env vars that link to kubernetes secrets?
k

Kevin Grismore

04/21/2023, 6:39 PM
when you create the work pool there will be a job template you can fill out, very similar to the kubernetes job block, where you can define things like the image and secrets and so on
👍 1
:upvote: 1
:thank-you: 1
b

Brett

04/21/2023, 7:21 PM
So, quick follow up question: Is it possible to point that to the kubernetes secretKeyRef ex:
{
"PYODBC_CONN_STR": {
"valueFrom": {
"secretKeyRef": {
"name": "prefect-env",
"key": "PYODBC_CONN_STR"
}
}
}
}
k

Kevin Grismore

04/21/2023, 7:27 PM
when you're creating your kubernetes work pool in the prefect UI, switch the base job template to 
advanced
and then you can edit it like a typical kubernetes job manifest, including for your env vars
👍 2
you should also be able to place that part in the environment variables portion of the default template
b

Brett

04/27/2023, 2:08 PM
Update: That json does not work, it expects the format to be 
{
  "key": "value",
  "ENVIRONMENT": "dev"
}
Even with that, passing in the right side ends up just being a string, and does not load from the kubernetes secrets. Or fails completely if you attempt to pass an object.
k

Kevin Grismore

04/27/2023, 3:15 PM
this is if you try to fill out the env var section in the ui?
b

Brett

04/27/2023, 3:15 PM
Correct.
k

Kevin Grismore

04/27/2023, 3:21 PM
what if you tried adding that json to the advanced job template for the work pool?
b

Brett

04/27/2023, 3:29 PM
I'm seeing a couple of places where there are 
{{ env }}
variables in that template. Could it be under the 
job_manifest/spec/template/spec/containers/{[env]}
? 
"job_manifest": {
      "kind": "Job",
      "spec": {
        "template": {
          "spec": {
            "containers": [
              {
       ------>> "env": "{{ env }}",
                "args": "{{ command }}",
                "name": "prefect-job",
                "image": "{{ image }}",
                "imagePullPolicy": "{{ image_pull_policy }}"
              }
            ],
k

Kevin Grismore

04/27/2023, 3:42 PM
I think it may have been the formatting of your json. go back to the default work pool template and try this as your env var 
{
  "name": "PYODBC_CONN_STR",
  "valueFrom": {
      "secretKeyRef": {
        "key": "PYODBC_CONN_STR",
        "name": "prefect-env"
      }
    }
  }
}
ahhhh okay I'm looking at the worker code
self.job_manifest["spec"]["template"]["spec"]["containers"][0]["env"] = [
     {"name": k, "value": v} for k, v in self.env.items()
 ]
what you said is correct. 
env
is expecting just 
"name": "value"
pairs.
👍 1
it seems like 
env
is inherited from 
prefect.workers.base.BaseJobConfiguration
, which defines 
env
like: 
class BaseJobConfiguration(BaseModel):
    ...
    env: Dict[str, Optional[str]] = Field(
        default_factory=dict,
        title="Environment Variables",
        description="Environment variables to set when starting a flow run.",
    )
I would try replacing the 
{{ env }}
part of the template, but given how this is written, I don't think that would work
b

Brett

04/27/2023, 5:07 PM
So it'll probably work the same way as the UI. it expects a string and only a string, and will fail if an object is used. Thanks for looking at that @Kevin Grismore! I'm going to just go with secret blocks. I was trying to avoid them so I didn't have another place I'd have to configure things!
#prefect-kubernetesJoin Slack
Powered by