https://prefect.io logo
Title
s

Samuel Hinton

02/24/2023, 2:41 AM
Hi team! I might be confused, or there might be an issue with the doco. Please see the screenshot. I’m trying to figure out what the difference is between a work pool and a work queue. The doco gives examples of starting an agent with no queue and pool of “my-pool”, but then talks about how this will great a “my-queue” work queue. I dont see how the two link up and have been assuming this is a doco error. Would anyone be able to confirm/deny this?
1
r

Ryan Peden

02/24/2023, 2:58 AM
I think it's an error. I believe it would only create
my-queue
inside
my-pool
if you ran
prefect agent start -p my-pool -q my-queue
. Work pools have to be created before you can use them, but if you ask for a queue that doesn't yet exist in the pool, it will be created for you automatically.
s

Samuel Hinton

02/24/2023, 2:59 AM
Gotcha
r

Ryan Peden

02/24/2023, 2:59 AM
If you don't specify a queue I believe it will just use the
default
queue in the work pool.
s

Samuel Hinton

02/24/2023, 2:59 AM
So my current use case is a few flows, some of which go to a dask worker, some which launch ECS tasks and dont need task
For that, I have two agents, one for a dask queue and one for a normal process queue. Should I update this to two pools? Im not quite sure on the appropriate/desired way to use pools vs queues
As another thing, the doco says “If the work pool does not exist, it will be created.” so to align that with your comment, does that mean its supposed to be talking about queues there as well, or will it autocreate both?
r

Ryan Peden

02/24/2023, 3:03 AM
Good question. I'm going to check on the docs, because what the docs say doesn't quite match what I see when using work pools from the CLI.
👍 1
As for whether to update to two pools...it depends on your needs. The nice thing about pools is that you can set a flow concurrency limit for the entire pool, and set a different priority for each queue within the pool to avoid a scenario where you've hit your concurrency limit, and high priority flows don't run because there are too many low priority flows queued up in front of them. That's not always needed, though. If your current setup works the way you need it to, I think it's fine to just leave both queues in the same pool.
s

Samuel Hinton

02/24/2023, 3:08 AM
Awesome, thanks for the insights 🙂
r

Ryan Peden

02/24/2023, 3:15 AM
You're welcome! 🙂
s

Samuel Hinton

02/24/2023, 3:16 AM
I hope you dont mind but Ive got a few more questions to drop i nthe community chat. Hope its not too much!
r

Ryan Peden

02/24/2023, 3:17 AM
Not at all! We always welcome more questions.