@Marvin how to setup in prefect.yaml, job_variables with resouce limit and requests?

To set up resource limits and requests, as well as configure

job_variables

in the

prefect.yaml

configuration file for Prefect 2.0, you'll need to be aware of how the configuration system works with respect to deployment and execution environments (as applicable). However, based on the search conducted through the documentation, I did not find a direct excerpt that specifically addresses configuring resource limits and requests in the

prefect.yaml

file. Typically, in Kubernetes, you would set resource limits and requests within the Kubernetes execution environment configuration to control CPU and memory allocation. I would expect a similar process in Prefect 2.x where you can specify these details within the execution environment or the job's configuration. For

job_variables

, based on the documentation, you can override environment variables specified in a work pool through the

job_variables

parameter. For example, when deploying a flow with Prefect CLI, you could use the

job_variables

to specify environment variables like below:

python
if __name__ == "__main__":
    get_repo_info.deploy(
        name="my-deployment-never-pull",
        work_pool_name="my-docker-pool",
        job_variables={"env": {"EXTRA_PIP_PACKAGES": "boto3"}},
        image="my-image:my-tag",
        push=False
    )

In this example, the environment variable

EXTRA_PIP_PACKAGES

denotes a special variable that Prefect uses to install additional Python packages at runtime. As for configuring these settings within

prefect.yaml

, you would typically define your deployment and runtime settings following the Prefect's configuration conventions. The

prefect.yaml

file should have sections corresponding to different Prefect components like

flows

,

deployments

,

logging

, etc., and you can add custom configuration there. Since the specific details about configuring resource requests and limits are not directly available in the excerpt returned, it might require delving deeper into Prefect's documentation or reaching out to their community or support channels for more precise information on this topic. If you want to get more accurate information on configuring

job_variables

and resource constraints in the

prefect.yaml

file, you might need to look for example configuration files or seek assistance from Prefect's documentation and support.

Hey Bruno, I recently worked on this so I thought I'd provide some example on top of Marvins help: In this example I use job_variables as a templating variable under definitions and reference it to job_variables with * (see this docs page for explaination of the * ) Then in job_variables, you define based on what work pools base job template is/override them. Hope this helps

name: ...
prefect-version: 2.14.9

definitions:
  job_variables:
    cpu1_memory1024: &cpu1_memory1024
      region: us-central1
      cpu: 1000m
      memory: 1024Mi
      env: ...


deployments:
  - name: ...
    description: ...
    tags: ["main", "{{ $IMAGE_TAG }}"]
    entrypoint: ...
    work_pool:
      name: ...
      work_queue_name: default
      job_variables: *cpu1_memory1024

Thank you @Jack P I will try that

ahh I am not super familiar with Docker Work pool default template. When I look in UI/under advanced, I don't see any options in configuration for resources/limit/cpu & memory 😕 but might be understanding incorrectly or looking in wrong spot

@Jack P It's Kubernetes, but the only config that works is inside work_pool config, Base Job Template. But with that, I can't define distinct resources, only If I create one work_pool for each flow

the issue here is the nesting

@Nate My base job template was exacly like that

"job_configuration": {
    "env": "{{ env }}",
    "name": "{{ name }}",
    "labels": "{{ labels }}",
    "command": "{{ command }}",
    "namespace": "{{ namespace }}",
    "job_manifest": {
      "kind": "Job",
      "spec": {
        "template": {
          "spec": {
            "containers": [
              {
                "env": [
                  {
                    "name": "VCD_CREDENTIAL",
                    "valueFrom": {
                      "secretKeyRef": {
                        "key": "vcdpre",
                        "name": "vcdpre"
                      }
                    }
                  },
                  {
                    "name": "S3_CREDENTIAL",
                    "valueFrom": {
                      "secretKeyRef": {
                        "key": "aws-secret",
                        "name": "aws-secret"
                      }
                    }
                  }
                ],
                "args": "{{ command }}",
                "name": "prefect-job",
                "image": "{{ image }}",
                "imagePullPolicy": "{{ image_pull_policy }}",
                "resources": {
                  "limits": {
                    "cpu": "500m",
                    "memory": "512Mi"
                  },
                  "requests": {
                    "cpu": "200m",
                    "memory": "256Mi"
                  }
                }
              }
            ],
            "completions": 1,
            "parallelism": 1,
            "restartPolicy": "Never",
            "serviceAccountName": "{{ service_account_name }}"
          }
        },
        "backoffLimit": 0,
        "ttlSecondsAfterFinished": "{{ finished_job_ttl }}"
      },
      "metadata": {
        "labels": "{{ labels }}",
        "namespace": "{{ namespace }}",
        "generateName": "{{ name }}-"
      },
      "apiVersion": "batch/v1"
    },
    "stream_output": "{{ stream_output }}",
    "cluster_config": "{{ cluster_config }}",
    "job_watch_timeout_seconds": "{{ job_watch_timeout_seconds }}",
    "pod_watch_timeout_seconds": "{{ pod_watch_timeout_seconds }}"
  }

combining with prefect.yaml

definitions:
  work_pools:
    my_docker_work_pool: &my_docker_work_pool
      name: k8s_pod
      work_queue_name: default
      job_variables:
        env:
          CREDENTIALZ: 'HI'
        resources:
          limits:
            cpu: 500m
            memory: 512Mi
          requests:
            cpu: 300m
            memory: 312Mi
      
# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: cloud_director_github
  version:
  tags: []
  description:
  entrypoint: flows/cloud_director_mpc/flow.py:kube_flow
  parameters: {}
  work_pool: *my_docker_work_pool
  schedule:

in the above you're treating

resources

as a top level key of

job_variables

, but it actually is nested like:

job_variables

: •

job_manifest

◦

spec

and so on so when we try to merge your deployment level overrides with the existing work pool, we likely discard / fail to merge if we find a key at an unexpected level i'd probably recommend making a

spec

yaml definition so you can do something like

definitions:
  work_pools:
    my_docker_work_pool: &my_docker_work_pool
      name: k8s_pod
      work_queue_name: default
      job_variables:
        env:
          CREDENTIALZ: 'HI'
        job_manifest:
          kind: Job
          spec: *my_spec_w_resource_reqs

@Nate Now it's working, but not following the top level key of the dict, finally the Pod is following the job_variables POD image

got it! thanks for sharing