@Marvin Can Prefect V3 workers start up and pull jobs based on tags? I see they have -p (pool) and -q (queue) but can they filter tasks by tag?

In Prefect V3, workers are designed to poll work pools for new runs to execute. They can be configured to poll from one or more queues within a work pool. However, there is no direct mention in the documentation of workers filtering tasks by tags when pulling jobs. Workers primarily use work pools and queues for managing and executing flow runs. For more detailed information on how workers operate, you can refer to the Prefect V3 workers documentation.

@Marvin Using queues, how can I have multiple priority queues for a single deployment? Is this possible? Say I have deployment_a and wanted a queue_a and queue_a_high where the queue_a_high is set as higher priority in the work_queues

In Prefect V3, you can indeed set up multiple priority queues for a single deployment. This is managed through work pools, where you can define different queues with varying priorities. Here's how you can conceptualize the setup: 1. Define Multiple Queues: You can create queues like

queue_a

and

queue_a_high

. 2. Set Priorities: Assign a higher priority to

queue_a_high

compared to

queue_a

. Priority determines the order of flow runs submitted for execution. If all flow runs can be executed without concurrency limitations, priority still determines the submission order. 3. Execution Order: The system will submit all work from the highest priority queue first (e.g.,

queue_a_high

) before moving to lower priority queues (e.g.,

queue_a

). For more detailed guidance, refer to the work pools documentation. This setup allows you to manage flow run execution more effectively, especially when dealing with urgent or critical tasks.

@Marvin How can I define the multiple queues for one deployment in my prefect.yaml?

To define multiple queues for a single deployment in your

prefect.yaml

file, you can specify different deployment configurations within the same file. Each deployment can be associated with a different work queue. Here's a basic structure of how you might set it up:

yaml
deployments:
  - name: deployment_a
    flow_name: my_flow
    work_pool:
      queue: queue_a
  - name: deployment_a_high
    flow_name: my_flow
    work_pool:
      queue: queue_a_high

In this example,

deployment_a

is associated with

queue_a

, and

deployment_a_high

is associated with

queue_a_high

. Each deployment configuration must have a unique

name

field, even if they reference the same flow. This allows you to manage different priority queues for the same deployment. For more details, you can refer to the Prefect YAML documentation.

@Marvin This creates multiple deployments

You're correct, the approach I described would indeed create multiple deployments. Prefect V3's current design does not support assigning multiple queues to a single deployment directly within a

prefect.yaml

configuration. Each queue requires a separate deployment configuration. If you need to manage priority within a single deployment context, you might consider using different work queues with separate deployments as a workaround. Each deployment can be prioritized by assigning different queue priorities, as previously discussed. If you have further questions or need additional guidance, feel free to ask.

@Marvin I have a deployment which has the status "Not Ready", does this just mean no worker is listening to it's queue?

The "Not Ready" status for a Prefect deployment typically indicates that no worker has polled its associated work queue in the last 60 seconds. This status suggests that the deployment is not currently able to execute because there are no active workers listening to its queue. Here's a brief summary of the relevant information: - Work Queue Status: A work queue is considered "READY" when a worker has polled it recently (within the last 60 seconds). If a queue hasn't been polled by a worker in that timeframe, it will show as "NOT_READY". - Deployment Status Changes: The status of a deployment can change from "ready" to "not ready" if the associated work pool or queue is not being actively polled by a worker. If you want to ensure your deployment is "Ready," make sure that there are workers actively polling the queues associated with your deployment. For more details, you can refer to the Prefect documentation on work pools.