https://prefect.io logo
#prefect-community
Title
# prefect-community
n

Nimesh Kumar

05/26/2023, 7:43 AM
Hi everyone, I have a question regarding the priority queue in the workpool So let suppose i have 2 queue (q1, q2) in a workpool (w) where q1 in a priority queue with highest priority. Now let say i have to multiple input (input1, input2 and so on) where "input 23" has the highest priority and need to processed as soon as it came to "q1" My question here is: 1. The scenario i mentioned above is okay ? 2. if input 16 come to q1 then all the inputs which are in q2 will be put on to hold till input 16 of q1 is processed ? Can anyone please help me on this.
c

Christopher Boyd

05/26/2023, 1:18 PM
Hi Nimesh, I think I understand your question, so I’ll try to reply. You have a high priority queue (queue 1), and a low priority queue (q2). If there is no concurrency or limitation in your work, AND both jobs can be submitted to infrastructure (local, docker, ECS, EKS, etc.) without Infrastructure being the bottleneck, then they will both execute roughly simultaneously. But lets say that’s not the case - you have a number of flow-runs that are in a
Scheduled
state and waiting to be run. Then, any work in queue 1, your high priority queue will take precedence. Once queue 1 is empty, it will proceed to queue 2, and begin processing work in queue 2 - unless / until more work is received on queue 1. Essentially, this means that ALL work in queue 1 will be completed and evaluated before submitting anythign in queue 2 or further. If anything new comes in on queue 1, that will be processed before continuing back on other queues pending work. This means if you continuously submit work to a high-priority queue, no work below would be scheduled
n

Nimesh Kumar

05/27/2023, 12:57 AM
Thanks a lot for this crisp explanation 🙏