@Marvin How should a work queue be configured when using the kubernetes worker type. e.g. to create a queue named limit-1 with a max concurrency of 1. I would like to create the queue with the Helm chart if possible

You can configure work queues with concurrency limits by using the Prefect Helm chart to deploy a worker. However, the available documentation snippets don't provide a direct answer about configuring work queues with a Helm chart. Concurrency settings are typically part of work queue configuration, rather than the Kubernetes deployment itself. For Helm chart details, you can consult the official Prefect Helm Chart documentation, but specific parameters for limiting concurrency on a work queue via Helm are not provided in the excerpts I found. It seems you may need to create the work queue separately from the Helm deployment and then ensure your Kubernetes workers are configured to pull from this queue. You would first create a work queue with a concurrency limit using the Prefect CLI or UI, and then set your Kubernetes workers to interact with this work queue. Use the Prefect CLI or UI to create a work queue with a concurrency limit, for example (not via Helm):

shell
prefect work-queue create limit-1 --concurrency-limit 1

Then configure your Kubernetes workers to pull from this queue. This configuration might be specified via environment variables or command-line arguments depending on your Helm chart configurations. Unfortunately, without the precise Helm values, I can't provide the exact configuration. Please check the Prefect Helm chart's

values.yaml

file or reach out to the Prefect community for specific details.

@Marvin So a work queue does not belong to a specific work pool? A queue can be consumed by one or many work pools?

Based on Prefect's documentation, work queues are indeed associated with work pools. Each work pool has a "default" queue that all work will be sent to by default. Additional queues can be added to a work pool, and each of these work queues can have its own concurrency limits. However, they are also subject to the global concurrency limit set on the work pool. The documentation explains that new queues can have fine-grained priority and concurrency, allowing for precision in controlling work delivery. Each work queue within a pool has a unique priority indicated by a positive integer, with lower numbers taking higher priority in the allocation of work. Importantly, each work queue's concurrency limit is still overseen by the work pool’s global concurrency limit, which cannot be exceeded. In essence, work queues can be seen as subdivisions within a work pool that specify more granular behavior regarding priority and concurrency. This system enables creating multiple queues within the same work pool for different purposes while managing overall work concurrency across all of them. For further details, you can reference this section in the Prefect documentation on work pools.

@Marvin So how does this cli command you provided

prefect work-queue create limit-1 --concurrency-limit 1

specify which work pool the queue is going to be associated with?

The Prefect documentation excerpts do not provide a specific CLI command for associating a newly created work queue directly with a specific work pool. The

prefect work-queue create

command given earlier is to create a work queue with specific settings, such as a concurrency limit, but it does not specify the work pool during creation. In Prefect, each work pool has a default queue associated with it. It seems that additional queues would be managed through the Prefect UI or some other mechanism that allows you to associate queues with specific work pools. If you need to associate a work queue with a specific work pool, it would be best to consult the Prefect documentation on work pools or reach out directly to the Prefect community or support. As of now, based on the available documentation, you would create the work queue separately and then perform additional steps within the Prefect UI or CLI to manage where the work queue applies. Additional queues can be added to a work pool, but the exact command to associate a work queue with a specific work pool is not made clear in the excerpts provided.

Cli --help is more use

prefect work-queue create --help
 --limit     -l      INTEGER  The concurrency limit to set on the queue. [default: None]                                                                                                                                                                                                                │
│ --tag       -t      TEXT     DEPRECATED: One or more optional tags. This option will be removed on 2023-02-23. [default: None]                                                                                                                                                                         │
│ --pool      -p      TEXT     The name of the work pool to create the work queue in. [default: None]                                                                                                                                                                                                    │
│ --priority  -q      INTEGER  The associated priority for the created work queue [default: None]                                                                                                                                                                                                        │
│ --help                       Show this message and exit.

The documentation excerpts provided do not include the syntax for specifying more than one queue when creating a work pool using the CLI. However, you've referenced the

--work-queue

option which suggests that it could be used to specify work queues for the worker to pull from. In many CLI tools, to specify multiple values for an option, you either repeat the option with each value or provide a comma-separated list. Without explicit documentation on the syntax, I would generally try one of these methods. For Prefect's CLI, if I were in your shoes, I would first try to specify multiple queues by repeating the

--work-queue

option with one queue name each time, like so:

shell
--work-queue queue1 --work-queue queue2

Alternatively, if that does not work, a comma-separated list is also common in command line tools:

shell
--work-queue queue1,queue2

However, since neither of these methods are confirmed by the current excerpts, I would recommend consulting the Prefect CLI's help text (

prefect work-pool create --help

) for the correct syntax or reaching out to Prefect's support or community for clarification. You may also want to review the work pools documentation for official guidance on this matter.

prefect worker start --name test --pool test --work-queue abc --work-queue xyz

appears to be the correct syntax but there does not appear to be anyway of checking the associated work-queues of a worker. The Prefect server UI shows nothing for the created work queues but the cli does list them correctly.

prefect:2.14.19-python3.11