https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> In Prefect 2.20, I am creating deploy...
# marvin-ai
f
@Marvin In Prefect 2.20, I am creating deployments using python script as follows:
Copy code
async with get_client() as client:
    await client.create_work_pool(
        work_pool=WorkPoolCreate(
            name=flow_specific_work_pool_name,
            type=worker.type,
            base_job_template=template,
            concurrency_limit=10,
        ),
        overwrite=True  # overwrite=True does not change the pool type
    )

    flow_id = await client.create_flow_from_name(flow_name)
    await client.create_deployment(
        flow_id=flow_id,
        name=flow_name,
        work_pool_name=flow_specific_work_pool_name,
        path=str(get_project_root()),
        entrypoint=f"{flow.__module__}:{flow_name}",
        parameters={}
    )
I want to create a 
prefect.yaml
file containing the configuration for each flow. Is there a way to reference this yaml file while creating the deployment?
m
✅ thought for 145.4 seconds
Short answer: not directly. In Prefect 2.20, 
client.create_deployment(...)
does not have a parameter to point at a 
prefect.yaml
file. That API expects the deployment fields to be passed explicitly, like you're doing now. So if you want to use 
prefect.yaml
, you generally have 2 options: * Use the CLI (
prefect deploy
/ older 2.x deployment workflows) to read 
prefect.yaml
* Parse the YAML yourself in Python and pass the values into 
client.create_deployment(...)
A 
prefect.yaml
is not automatically consumed by 
client.create_deployment(...)
. Example of what that means in practice:
Copy code
yaml
deployments:
  - name: my-flow
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: my-pool
Then in Python, you'd need to do something like:
Copy code
python
import yaml

with open("prefect.yaml", "r") as f:
    config = yaml.safe_load(f)

deployment_cfg = config["deployments"][0]

await client.create_deployment(
    flow_id=flow_id,
    name=deployment_cfg["name"],
    work_pool_name=deployment_cfg["work_pool"]["name"],
    entrypoint=deployment_cfg["entrypoint"],
    path=deployment_cfg.get("path"),
    parameters=deployment_cfg.get("parameters", {}),
)
A couple of notes: * 
prefect.yaml
is mainly intended for deployment configuration used by CLI-based deployment workflows * Your current 
client.create_deployment(...)
approach is a lower-level API, so it won't automatically resolve that file * In Prefect 3.x, the recommended deployment APIs changed significantly, so if you're planning ahead, it may be worth designing around the newer deployment patterns If you'd like, I can help you with either of these: 1. a sample 
prefect.yaml
structure for multiple flows in Prefect 2.20, or 2. a Python helper that loads one deployment definition from 
prefect.yaml
and calls 
create_deployment(...)
for each flow.
f
@Marvin can you create a sample 
prefect.yaml
structure for multiple flows in Prefect 2.20 where each flow has a different resource requirement and limit? All of them are of type k8s
m
✅ thought for 15.0 seconds
Yes — here’s a good Prefect 2.20-style 
prefect.yaml
pattern for multiple flows, where each deployment targets a Kubernetes work pool but uses different Kubernetes resource requests/limits via 
job_variables
. Important caveat: in Prefect 2.x, resource settings are usually applied per deployment through the work pool’s base job template and overridden with 
job_variables
on each deployment. A sample 
prefect.yaml
could look like this:
Copy code
yaml
name: my-project

build: []
push: []
pull: []

deployments:
  - name: small-flow
    version: "1.0.0"
    tags:
      - k8s
      - small
    description: Small resource flow
    entrypoint: flows/small_flow.py:small_flow
    parameters: {}
    work_pool:
      name: my-k8s-work-pool
      job_variables:
        image: my-registry/my-image:latest
        namespace: default
        cpu_request: "500m"
        cpu_limit: "1"
        memory_request: "512Mi"
        memory_limit: "1Gi"

  - name: medium-flow
    version: "1.0.0"
    tags:
      - k8s
      - medium
    description: Medium resource flow
    entrypoint: flows/medium_flow.py:medium_flow
    parameters: {}
    work_pool:
      name: my-k8s-work-pool
      job_variables:
        image: my-registry/my-image:latest
        namespace: default
        cpu_request: "1"
        cpu_limit: "2"
        memory_request: "1Gi"
        memory_limit: "2Gi"

  - name: large-flow
    version: "1.0.0"
    tags:
      - k8s
      - large
    description: Large resource flow
    entrypoint: flows/large_flow.py:large_flow
    parameters: {}
    work_pool:
      name: my-k8s-work-pool
      job_variables:
        image: my-registry/my-image:latest
        namespace: default
        cpu_request: "2"
        cpu_limit: "4"
        memory_request: "4Gi"
        memory_limit: "8Gi"
A few important points: * 
work_pool.name
is the same for all deployments here, since all are using your Kubernetes pool * 
job_variables
is where you override per-deployment infrastructure values * The actual variable names you can use in 
job_variables
depend on your Kubernetes work pool’s base job template * If your base job template does not define variables like 
cpu_request
, 
cpu_limit
, etc., these keys won’t do anything automatically So for this to work, your K8s work pool’s base job template needs to expose variables that map into the Kubernetes job spec. For example, the base job template might include placeholders like:
Copy code
json
{
  "variables": {
    "properties": {
      "cpu_request": {
        "type": "string",
        "default": "500m"
      },
      "cpu_limit": {
        "type": "string",
        "default": "1"
      },
      "memory_request": {
        "type": "string",
        "default": "512Mi"
      },
      "memory_limit": {
        "type": "string",
        "default": "1Gi"
      }
    }
  },
  "job_configuration": {
    "container_resources": {
      "requests": {
        "cpu": "{{ cpu_request }}",
        "memory": "{{ memory_request }}"
      },
      "limits": {
        "cpu": "{{ cpu_limit }}",
        "memory": "{{ memory_limit }}"
      }
    }
  }
}
So the overall pattern is: 1. Create one Kubernetes work pool 2. Define resource-related variables in its base job template 3. In 
prefect.yaml
, set different 
job_variables
per deployment If you want, I can also give you: * a complete example Kubernetes work pool base job template for Prefect 2.20, or * a matching Python script that reads this 
prefect.yaml
and creates all deployments programmatically.
f
@Marvin If I set one work pool with 2 queues, each having its own worker, then does queue priority still make any difference?
m
✅ thought for 127.0 seconds
In your setup: mostly no. If each queue has its own dedicated worker, and each worker polls only that queue, then queue priority does not really matter. Why: * Queue priority is used when a worker is deciding which queue to pull from first * If Worker A only watches 
queue-a
and Worker B only watches 
queue-b
, there’s no cross-queue choice to make * Each worker just pulls runs from its own queue So in this setup:
Copy code
text
work pool
├── queue-a -> worker-a
└── queue-b -> worker-b
priority between 
queue-a
and 
queue-b
has no practical effect. It does matter when: * one worker polls multiple queues, or * workers poll the whole pool and can pick up work from different queues, or * concurrency constraints mean not everything can be submitted at once Then priority helps Prefect decide which queues get precedence. So the rule of thumb is: * Dedicated worker per queue → priority is basically irrelevant * Shared workers across queues → priority matters Related docs: * Queue priority in work pools * Work pools and work queues If you want, I can also help you think through whether multiple queues + multiple workers is actually the best design for expressing different Kubernetes resource classes, versus using one queue and per-deployment 
job_variables
.
Open in Slack
PreviousNext
Powered by