HI everyone , I am running into a exception when I am running a job in kubernetes. The error message...

Deceivious

04/20/2023, 8:47 AM

HI everyone , I am running into a exception when I am running a job in kubernetes. The error message is:

Copy code

"jobs.batch is forbidden: User \"system:serviceaccount:prefect2-worker-dev:prefect-worker\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"

How do I change the jobs from being created in default namespace and change it to some other name space?

Deceivious

04/20/2023, 8:48 AM

For more context, I deploy prefect flows using the Prefect Python library.

Copy code

deployment = Deployment.build_from_flow(
        flow=flow,
        name=deployment_name,
        schedule=schedule,
        tags=tags,
        is_schedule_active=is_schedule_active,
        work_pool_name=work_pool_name,
        work_queue_name=work_queue_name,
        infrastructure=KubernetesJob(
            namespace=config.get("INFRA", "KUBERNETES_NAMESPACE") # THIS IS NOT DEFAULT
        ),
        infra_overrides=generate_kubernetes_overrides(
            manifest, environment, mem_limit_gb
        ),
        path=DOCKER_WORK_DIR,
    )
deployment.apply()

And I have deployed prefect workers using hellm from the

prefect-helm

github.

redsquare

04/20/2023, 8:54 AM

try adding the namespace to infra_overrides

redsquare

04/20/2023, 8:54 AM

infra_overrides["namespace"]=config.get("INFRA", "KUBERNETES_NAMESPACE")

redsquare

04/20/2023, 9:01 AM

or maybe it is simply the same as this https://prefect-community.slack.com/archives/C048SVCEFF0/p1675137176802559?thread_ts=1675116099.790469&cid=C048SVCEFF0

Deceivious

04/20/2023, 9:03 AM

Unsure if overriting the existing namespace with the same namspace will make a difference but m testing that out.

redsquare

04/20/2023, 9:04 AM

yeah, the other link more likely

Deceivious

04/20/2023, 12:01 PM

Unsure if that is the issue as well. Helm creates service qaccounts by default. Its just that the jobs are being spawns in the wrong name space

redsquare

04/20/2023, 12:04 PM

clusterrolebinding for the sa?

Deceivious

04/20/2023, 12:07 PM

The config is from https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-worker/values.yaml

redsquare

04/20/2023, 12:12 PM

sure - not used the helm chart

Deceivious

04/20/2023, 12:13 PM

Yes , I deployed previously without the helm charts and they were working - now that i am using helm charts, ive started having these issues

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.