Join Slack

What is the format for adding multiple pip packages for a KubernetesJob?

# ask-community

Devin Flake

03/02/2023, 9:50 PM

What is the format for adding multiple pip packages for a KubernetesJob?

Devin Flake

03/02/2023, 9:51 PM

this isn't working for me:

Copy code

k8s_job = KubernetesJob(
    namespace="prefect-server",
    image="prefecthq/prefect:2.8.2-python3.11",
    env={"EXTRA_PIP_PACKAGES": "adlfs prefect-dask"},
    job=KubernetesJob.job_from_file("k8s_flow_run_job_manifest.yaml"),
)
k8s_job.save("prod", overwrite=True)

redsquare

03/02/2023, 10:09 PM

you need to use infra_overrides infra_overrides['env'] = { 'EXTRA_PIP_PACKAGES': 'bla bla'}

👍 1

Zanie

03/02/2023, 10:37 PM

Hm, just

env

should be working

Zanie

03/02/2023, 10:37 PM

Why would you need to use infra overrides?

redsquare

03/02/2023, 10:38 PM

env is for environment variables

Zanie

03/02/2023, 10:38 PM

This is an environment variable that is read by our entrypoint

redsquare

03/02/2023, 10:38 PM

https://docs.prefect.io/api-ref/prefect/infrastructure/#prefect.infrastructure.KubernetesJob

Zanie

03/02/2023, 10:39 PM

i.e. https://github.com/PrefectHQ/prefect/blob/main/scripts/entrypoint.sh#L11

redsquare

03/02/2023, 10:42 PM

hmmm that didnt used to work

redsquare

03/02/2023, 10:47 PM

https://github.com/PrefectHQ/prefect/pull/6581

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.