Here's an example of a

prefect.yaml

configuration file that defines a workpool with three work queues of different priorities:

yaml
workspaces:
  default:
    workpool:
      name: "my-workpool"
      type: "process"
      work_queues:
        - name: "critical"
          priority: 100  # Higher number = higher priority
        - name: "important"
          priority: 50
        - name: "routine"
          priority: 1

deployments:
  - name: default
    version: 0.1.0
    tags: []
    description: "Default deployment"
    schedule: {}
    flow_name: null
    entrypoint: null
    parameters: {}
    work_queue_name: routine  # Default to routine queue

A few key points about this configuration: 1. The workpool is set to use the "process" infrastructure type, which runs flows as local processes. You can change this to other types like "kubernetes" or "docker" based on your needs. 2. The priority values (100, 50, 1) determine which work queue's flows run first when resources are limited. Higher numbers mean higher priority. 3. When creating deployments, you can specify which work queue to use via the

work_queue_name

field. If not specified, flows will use the default work queue. To use different queues in your deployments, you would modify the

work_queue_name

field:

yaml
deployments:
  - name: urgent-task
    work_queue_name: critical
    # ... other deployment settings ...

  - name: standard-task
    work_queue_name: routine
    # ... other deployment settings ...

When you start your worker, it will automatically handle tasks from all three queues according to their priorities.

Here's the modified

prefect.yaml

that includes different concurrency limits for each work queue:

yaml
workspaces:
  default:
    workpool:
      name: "my-workpool"
      type: "process"
      work_queues:
        - name: "critical"
          priority: 100
          concurrency_limit: 1  # Process critical tasks one at a time
        - name: "important"
          priority: 50
          concurrency_limit: 3  # Allow up to 3 important tasks simultaneously
        - name: "routine"
          priority: 1
          concurrency_limit: 5  # Allow up to 5 routine tasks simultaneously

deployments:
  - name: default
    version: 0.1.0
    tags: []
    description: "Default deployment"
    schedule: {}
    flow_name: null
    entrypoint: null
    parameters: {}
    work_queue_name: routine

The

concurrency_limit

settings mean: - Critical queue: Only one task runs at a time (good for resource-intensive operations) - Important queue: Up to 3 tasks can run simultaneously - Routine queue: Up to 5 tasks can run simultaneously This configuration allows you to both prioritize tasks and control resource usage by limiting how many flows can run concurrently in each queue. The worker will automatically respect both the priority and concurrency settings when processing tasks. Remember that these concurrency limits are per work queue - the total number of concurrent tasks that could be running is the sum of all concurrency limits (in this case, 1 + 3 + 5 = 9 maximum concurrent tasks).

I'll help you modify your definitions to include multiple work queues while maintaining your existing Kubernetes configuration. Here's how you can expand your current setup:

yaml
definitions:
    work_pools:
        k8_default: &k8_default
            name: k8
            type: kubernetes
            work_queues:
                - name: critical
                  priority: 100
                  concurrency_limit: 1
                  job_variables:
                    request_memory: "1Gi"
                    image_pull_policy: "Always"
                    nodeSelector:
                      node-type: static
                - name: important
                  priority: 50
                  concurrency_limit: 3
                  job_variables:
                    request_memory: "0.5Gi"
                    image_pull_policy: "Always"
                    nodeSelector:
                      node-type: static
                - name: routine
                  priority: 1
                  concurrency_limit: 5
                  job_variables:
                    request_memory: "0.5Gi"
                    image_pull_policy: "Always"
                    nodeSelector:
                      node-type: static

Key changes made: 1. Replaced the single

work_queue_name

with a

work_queues

list 2. Each queue can have its own

job_variables

- I kept them the same except for the critical queue where I increased the memory request as an example 3. Added priority and concurrency limits for each queue 4. Added

type: kubernetes

explicitly (though it would be inferred from the job variables) Now when creating deployments, you can specify which queue to use:

yaml
deployments:
  - name: my-critical-deployment
    work_queue_name: critical
    # ... other deployment settings ...

  - name: my-routine-deployment
    work_queue_name: routine
    # ... other deployment settings ...

Each queue will maintain the Kubernetes-specific configurations you had before, but now you have the flexibility to route different deployments to different queues with different priorities and concurrency limits.

Ah, I see you're using YAML anchors! Here's how you can structure your work queues with anchors to allow for reference and reuse:

yaml
definitions:
    work_pools:
        k8_default: &k8_default
            name: k8
            type: kubernetes
            work_queues:
                - name: critical
                  &k8_critical
                  priority: 100
                  concurrency_limit: 1
                  job_variables: &k8_job_vars
                    request_memory: "0.5Gi"
                    image_pull_policy: "Always"
                    nodeSelector:
                      node-type: static
                - name: important
                  &k8_important
                  priority: 50
                  concurrency_limit: 3
                  job_variables: *k8_job_vars  # Reuses the job variables defined above
                - name: routine
                  &k8_routine
                  priority: 1
                  concurrency_limit: 5
                  job_variables: *k8_job_vars  # Reuses the job variables defined above

In this configuration: -

&k8_default

is the anchor for the entire workpool configuration -

&k8_critical

,

&k8_important

, and

&k8_routine

are anchors for each queue configuration -

&k8_job_vars

defines the common job variables -

*k8_job_vars

references the common job variables (note the asterisk instead of ampersand) You can then reference these anchors elsewhere in your YAML file using the

*

operator. For example:

yaml
deployments:
  - name: high-priority-job
    work_queue_name: critical
    job_configuration: *k8_critical

  - name: routine-job
    work_queue_name: routine
    job_configuration: *k8_routine

This approach allows you to: 1. Define common configurations once and reuse them 2. Override specific values when needed 3. Maintain consistency across similar configurations 4. Reduce duplication in your YAML file Remember that YAML anchors (

&

) define a reusable block, while aliases (

*

) reference those blocks. The ampersand (

&

) creates the anchor, and the asterisk (

*

) uses it.

No, this structure won't work. The

work_queues

in Prefect's configuration needs to be a list, not a dictionary. Here's the corrected version:

yaml
definitions:
    work_pools:
        k8_default: &k8_default
            name: k8
            work_queues:
                - &k8_critical
                  name: critical
                  priority: 100  # Higher number = higher priority
                  concurrency_limit: 1
                  job_variables:
                    request_memory: "1Gi"
                    image_pull_policy: "Always"
                    nodeSelector:
                      node-type: static
                - &k8_important
                  name: important
                  priority: 50
                  concurrency_limit: 3
                  job_variables:
                    request_memory: "0.5Gi"
                    image_pull_policy: "Always"
                    nodeSelector:
                      node-type: static
                - &k8_routine
                  name: routine
                  priority: 1
                  concurrency_limit: 5
                  job_variables:
                    request_memory: "0.5Gi"
                    image_pull_policy: "Always"
                    nodeSelector:
                      node-type: static
                - &k8
                  name: k8
                  priority: 25
                  job_variables:
                    request_memory: "0.5Gi"
                    image_pull_policy: "Always"
                    nodeSelector:
                      node-type: static

Key changes made: 1. Changed the

work_queues

structure from a dictionary to a list (using

-

for each item) 2. Fixed the indentation for

nodeSelector

values 3. Adjusted the priorities to follow the "higher number = higher priority" convention 4. Maintained all your anchor tags (

&k8_critical

,

&k8_important

, etc.) Also, note that I reversed your priority numbers because in Prefect, higher numbers indicate higher priority. So if you want critical to be the highest priority, it should have the highest number. The corrected priority order is now: - critical: 100 (highest priority) - important: 50 - k8: 25 - routine: 1 (lowest priority) This structure will work properly with Prefect's workpool configuration system while maintaining your ability to reference these configurations using the YAML anchors.