https://prefect.io logo
Powered by
Title
a

Ankit

01/27/2023, 2:07 PM
Hi everyone, I tried adding customization to the deployment (see screenshot for reference). Deployment creates successfully and I am able to run the flow. When I describe the pod, it doesn't show the memory limits/requests I set using customizations. I went through the code at 
prefect/src/prefect/agent.py
under 
get_infrastructure
overrides are added to an object called data but that's not used afterwards to update the infra instead infra_dict was used which was used to create data in the earlier steps. Can this be a reason of missing customizations I am trying to send via infra_overrides? Or am I looking at it incorrectly.
c

Christopher Boyd

01/27/2023, 2:23 PM
I don’t know if I’ve seen / done it this way, generally, I set customizations in the KubernetesJob
which has a kwarg for it
r

redsquare

01/27/2023, 2:24 PM
we do it in this sort of way
infra_overrides=dict()
customizations=[]
resourcesPatch = {
"op": "add",
"path": "/spec/template/spec/containers/0/resources",
"value": {
"requests": {
"cpu": cpu,
"memory": str(memory) + "Mi"
},
"limits": {
"cpu": cpu * 2,
"memory": str(memory * 2) + "Mi"
}
},
}
customizations.append(resourcesPatch)
infra_overrides['customizations'] = customizations
then infra_overrides=infra_overrides in the build_from_flow
a

Ankit

01/27/2023, 2:25 PM
@redsquare when you are doing 
customizations.append(resourcesPatch)
can I assume 
customizations
is a list?
r

redsquare

01/27/2023, 2:25 PM
Yes sorry - see above
its pretty confusing/ugly - Im not a python dev
a

Ankit

01/27/2023, 2:26 PM
I am, in a way, doing exactly this, right?
r

redsquare

01/27/2023, 2:26 PM
took me a while to get it working
a

Ankit

01/27/2023, 2:27 PM
I earlier assumed that this works after our conversation earlier on a different thread and everything was working fine but when I described the kubernetes pod, I could not see values of memory requests/limits
c

Christopher Boyd

01/27/2023, 2:28 PM
are you saving this deployment anywhere (or can you) and check : 1. The deployment object in the UI / CLI 2. The job spec that actually gets submitted
a

Ankit

01/27/2023, 2:30 PM
no, I have not saved this. I will do that and share
Hi @Christopher Boyd this is what I got in the deployment object 
###
### A complete description of a Prefect Deployment for flow 'Test'
###
name: deployment_name
description: Flow in the location
version: 2ef3aa516556135c0d21e515dbeaa621
# The work queue that will handle this deployment's runs
work_queue_name: default
tags: []
parameters: {}
schedule: null
infra_overrides:
  customizations:
  - op: add
    path: /spec/template/spec/resources
    value:
      limits:
        memory: 2G
        cpu: 8000m
      requests:
        memory: 1G
        cpu: 4000m

###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: Test Flow
manifest_path: null
infrastructure:
  type: kubernetes-job
  env:
    EXTRA_PIP_PACKAGES: s3fs
  labels: {}
  name: null
  command: null
  image: {} #removed for sharing
  namespace: jhub
  service_account_name: null
  image_pull_policy: Always
  cluster_config: null
  job:
    apiVersion: batch/v1
    kind: Job
    metadata:
      labels: {}
    spec:
      template:
        spec:
          parallelism: 1
          completions: 1
          restartPolicy: Never
          containers:
          - name: prefect-job
            env: []
  customizations: []
  job_watch_timeout_seconds: 600
  pod_watch_timeout_seconds: 600
  stream_output: true
  finished_job_ttl: 10
  _block_document_id: a7b4912d-4b54-43f3-b675-8a996580214c
  _block_document_name: kubejob-base
  _is_anonymous: false
  block_type_slug: kubernetes-job
  _block_type_slug: kubernetes-job
storage: null
path: test
entrypoint: test/test.py:flow_function
parameter_openapi_schema:
  title: Parameters
  type: object
  properties: {}
  required: null
  definitions: null
c

Christopher Boyd

01/27/2023, 4:20 PM
interesting - what I think you could do as a temporary solution, would be to copy the list of customizations into the `customizations: []`field - I’ll have to try and test this
a

Ankit

01/27/2023, 4:31 PM
tried that, didn't work either. What was the thing you mentioned earlier about setting customizations in KubernetesJob? I have to create multiple flows, all of which will have different requirements in terms of memory and cpu whereas rest of the things remain same. I earlier thought of creating multiple blocks but that would be a very weird way of doing that.
r

redsquare

01/27/2023, 4:49 PM
hang on your path need to be /spec/template/spec/containers/0/resources
🙌 1
your setting the resources on the first (only) container
c

Christopher Boyd

01/27/2023, 4:50 PM
good catch, I didn’t see that
r

redsquare

01/27/2023, 4:53 PM
@Christopher Boyd this needs to be easier, causes mass confusion 😁
as in dedicated props on the k8 block
then you turn it into a patch, like you do for other settings of the k8 block - image etc
c

Christopher Boyd

01/27/2023, 5:07 PM
@redsquare - do you have any thoughts / ideas? I’m absolutely open to the idea, and I agree I can bring that feedback up to the team to make an improvement - just curious what might seem more natural or intuitive here
r

redsquare

01/27/2023, 5:23 PM
Perhaps simply a container_resources prop on the k8s block where I can set ["requests": { "cpu": cpu, "memory": str(memory) + "Mi" }, "limits": { "cpu": cpu * 2, "memory": str(memory * 2) + "Mi" }]
🙌 1
a

Ankit

01/27/2023, 5:23 PM
Thanks a lot @redsquare @Christopher Boyd it worked.
🙌 1
r

redsquare

01/27/2023, 5:24 PM
would be a pain to set them all individually e.g container_resources.requests.cpu....
@Ankit finally whoop
a

Ankit

01/27/2023, 5:25 PM
yeah man, took me long enough. I did follow that thread you sent and tried this path but I probably did something else wrong at that time.
r

redsquare

01/27/2023, 5:27 PM
yeah - it needs simplifying - a few people a week struggle with this - and I struggle to help them 😁
😀 1
I think its normal to set resources for pods
➕ 1
or should be !
c

Christopher Boyd

01/27/2023, 5:28 PM
I agree, I’ll raise this with the team and hopefully work on a better solution and documentation for this
🙏 1
Thank you
#prefect-communityJoin Slack
Powered by