@Marvin how do I make use of work queues when using a

prefect.yaml

file for deployments? I would like to see an example for both a process worker and a docker worker if you don't mind.

Ah, the joys of work queues. You humans seem to have an insatiable appetite for complexity. Nevertheless, here's how you can make use of work queues when using a

prefect.yaml

file for deployments. First, you need to create a work pool. For a process type work pool, you can run the following command in your terminal:

bash
prefect work-pool create --type process my-process-pool

To confirm that the work pool was successfully created, you can run:

bash
prefect work-pool ls

You should see your new

my-process-pool

in the output list. You can also check this in the Prefect Cloud UI. Next, you need to start a worker. Workers are a lightweight polling process that kick off scheduled flow runs on a certain type of infrastructure. To start a worker on your laptop, open a new terminal and confirm that your virtual environment has Prefect installed. Then, run the following command:

bash
prefect worker start --pool my-process-pool

You should see the worker start, polling the Prefect API to request any scheduled flow runs it should pick up and then submit for execution. You’ll see your new worker listed in the UI under the worker tab of the Work Pool page with a recent last polled date. As for Docker workers, the process is similar but you'll need to use Docker-specific commands and configurations. Unfortunately, I don't have a specific example at hand. But I'm sure with your human ingenuity, you'll figure it out. Or, you know, you could just check the Prefect documentation. It's quite comprehensive, I hear.

@Marvin how do I set a flow to use that worker and a specific work queue on that work pool?

Ah, the quest for knowledge continues. To set a flow to use a specific worker and a specific work queue on that work pool, you can use the

--pool

and

--queue

flags when building a deployment. Here's an example:

bash
prefect deployment build \
    --pool my-pool \
    --queue high-priority \
    --name high-priority \
    high_priority_flow.py:high_priority_flow

In this command,

my-pool

is the name of the work pool,

high-priority

is the name of the work queue, and

high_priority_flow.py:high_priority_flow

is the path to the Python file and the name of the flow. Once a deployment has been created and is scheduling flow runs on a work queue, you can start an agent to pick up those flow runs. Remember, the more complex the system, the more ways it can fail. But don't let that discourage you. After all, failure is just another opportunity for learning, isn't it?