https://prefect.io
Join Slack
Hi team! I might be confused, or there might be an issue with the doco. Please see the screenshot. ...
# ask-community
s
Hi team! I might be confused, or there might be an issue with the doco. Please see the screenshot. I’m trying to figure out what the difference is between a work pool and a work queue. The doco gives examples of starting an agent with no queue and pool of “my-pool”, but then talks about how this will great a “my-queue” work queue. I dont see how the two link up and have been assuming this is a doco error. Would anyone be able to confirm/deny this?
✅ 1
r
I think it's an error. I believe it would only create 
my-queue
inside 
my-pool
if you ran 
prefect agent start -p my-pool -q my-queue
. Work pools have to be created before you can use them, but if you ask for a queue that doesn't yet exist in the pool, it will be created for you automatically.
s
Gotcha
r
If you don't specify a queue I believe it will just use the 
default
queue in the work pool.
s
So my current use case is a few flows, some of which go to a dask worker, some which launch ECS tasks and dont need task
For that, I have two agents, one for a dask queue and one for a normal process queue. Should I update this to two pools? Im not quite sure on the appropriate/desired way to use pools vs queues
As another thing, the doco says “If the work pool does not exist, it will be created.” so to align that with your comment, does that mean its supposed to be talking about queues there as well, or will it autocreate both?
r
Good question. I'm going to check on the docs, because what the docs say doesn't quite match what I see when using work pools from the CLI.
👍 1
As for whether to update to two pools...it depends on your needs. The nice thing about pools is that you can set a flow concurrency limit for the entire pool, and set a different priority for each queue within the pool to avoid a scenario where you've hit your concurrency limit, and high priority flows don't run because there are too many low priority flows queued up in front of them. That's not always needed, though. If your current setup works the way you need it to, I think it's fine to just leave both queues in the same pool.
s
Awesome, thanks for the insights 🙂
r
You're welcome! 🙂
s
I hope you dont mind but Ive got a few more questions to drop i nthe community chat. Hope its not too much!
r
Not at all! We always welcome more questions.
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by