Hi! I’m trying to leverage Kubernetes for Prefect. Our K8s agent is up and is receiving flow runs pr...

Shaoyi Zhang

09/21/2021, 12:10 AM

Hi! I’m trying to leverage Kubernetes for Prefect. Our K8s agent is up and is receiving flow runs properly. However, the K8s job creation is failing due to

Copy code

Error creating: pods "prefect-job-138dbd9d-v6rq5" is forbidden: failed quota: xxxxx-request-quota: must specify requests.cpu,requests.memory

although I specified the request limit with

Copy code

env:
.....
- name: JOB_MEM_REQUEST
  value: "2048Mi"
- name: JOB_MEM_LIMIT
  value: "4096Mi"
- name: JOB_CPU_REQUEST
  value: "100m"
- name: JOB_CPU_LIMIT
  value: "200m"

This is the Prefect source code if I understand correctly and I’m using

prefect:0.15.5-python3.6

. Any suggestions?

Kevin Kho

09/21/2021, 1:37 AM

~~Are you confident it should be “100m” instead of “100Mi”?~~

Kevin Kho

09/21/2021, 1:42 AM

will check with the team

Kevin Kho

09/21/2021, 1:45 AM

wait! that source code is for environments. what you want is this method because this is the one for RunConfigs. Environments are for below 0.14.0

Kevin Kho

09/21/2021, 1:46 AM

And that method does not do the same operation to pull those environment variables. it only grabs them through the RunConfig here

Shaoyi Zhang

09/21/2021, 4:02 PM

Thank you! For anyone encounter this issue, you need to specify cpu_request, cpu_limit, memory_request, memory_limit in your K8s run config.

Copy code

flow.run_config = KubernetesRun(
    labels=["dev"],
    env={"test": "hahaha"},
    image="xxxxxxx",
    cpu_limit="200m",
    cpu_request="100m",
    memory_limit="4096Mi",
    memory_request="1048Mi"
)

👍 1

5 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.