Is there a preferred way of installing requirement...
# prefect-kubernetesj
John Mizerany
05/30/2023, 3:21 PMIs there a preferred way of installing requirements for a flow onto a kubernetes worker deployed using helm?
✅ 1
j
Jamie Zieziula
05/30/2023, 3:27 PMI believe the preferred method would be to build a custom docker image that has your requirements installed on it
z
Zanie
05/30/2023, 3:28 PM
An alternative is to set the
EXTRA_PIP_PACKAGESj
John Mizerany
05/30/2023, 3:37 PMIf I went the custom docker image route how would that affect us using the helm chart for deploying? The extra_pip_packages does sound a little easier since we typically don’t have that many extras to install for our flows, but the increase in startup time for the jobs does sound not ideal
z
Zanie
05/30/2023, 3:37 PM
Do you need the requirements to be available for the flow run pod or the worker pod?
j
John Mizerany
05/30/2023, 3:38 PMThe only pod that is deployed in our K8 cluster is the worker pod
John Mizerany
05/30/2023, 3:40 PMUnless there should also be a flow run pod? I just used the helm chart to deploy and when I go to view the pods running in our namespace I only see the worker pod
z
Zanie
05/30/2023, 3:40 PM
The worker creates a job / pod for each flow run
✅ 1
Zanie
05/30/2023, 3:42 PM
Anyway you probably want the requirements in the flow run pod which means you can just set the image on your flow’s deployment — no need for changing anything with the helm chart.
j
John Mizerany
05/30/2023, 3:42 PMGotcha. Okay that makes sense! Thank you so much
John Mizerany
05/31/2023, 3:08 PM@Zanie I tried to use the
EXTRA_PIP_PACKAGESworker.extraEnvVars.EXTRA_PIP_PACKAGES Copy code
│ Error: YAML parse error on prefect-worker/templates/deployment.yaml: error converting YAML to JSON: yaml: line 67: did not find expected '-' indicatorenv.EXTRA_PIP_PACKAGESj
Jamie Zieziula
05/31/2023, 3:13 PMyou’ll need to specify it with something like this:
Copy code
worker.extraEnvVars[0].name = EXTRA_PIP_PACKAGES
worker.extraEnvVars[0].value = PACKAGE_NAMEj
John Mizerany
05/31/2023, 3:15 PMAhhhhhh gotcha! So in theory I could do
Copy code
set {
name = "worker.extraEnvVars[0]"
value = "prefect-slack==0.1.2"
}j
Jamie Zieziula
05/31/2023, 3:51 PMhm I’m not sure that would work — I think this would work:
Copy code
resource "helm_release" "prefect" {
repository = "<https://prefecthq.github.io/prefect-helm>"
chart = "prefect-worker"
version = "2023.05.25"
name = "prefect-worker"
values = [<<-EOF
worker:
extraEnvVars:
- name: EXTRA_PIP_PACKAGES
value: prefect-slack==0.1.2
EOF
]
}👀 1
j
John Mizerany
05/31/2023, 5:14 PMThat deployed, but I have noticed another issue. The worker-pod definitely has that environment variable now, but when it goes to create flow run pods those do not have the same environment variable (EXTRA_PIP_PACKAGES don’t exist in the created flow run pod). So I don’t believe it is installing that package on flow execution
j
Jamie Zieziula
05/31/2023, 5:29 PMoh - if you want it for your flow pods you’ll need to set that env var in the infra_overrides section of either your work-pool or your deployment.
Jamie Zieziula
05/31/2023, 5:30 PMunder your work pool definition
Jamie Zieziula
05/31/2023, 5:34 PMor under your deployments.yaml
Copy code
deployments:
- name: Deployment Name
version: null
tags: []
description: null
schedule: {}
flow_name: null
entrypoint: path/to/flow.py:flow
parameters: {}
work_pool:
name: kubernetes
job_variables:
env:
EXTRA_PIP_PACKAGES: prefect-slack==0.1.2🙌 1
j
John Mizerany
05/31/2023, 5:40 PMYes that was it! Thank you so much
John Mizerany
05/31/2023, 5:41 PMApologies for all my questions, my brain is still working between Prefect 1.0 and 2.0 🙂 but I think I can take the rest for here 🚀
j
Jamie Zieziula
05/31/2023, 5:53 PMhaha no worries at all! happy to help
🙌 1
33 Views