3 is just a random number i took.
Please allow me to explain my usecase
Let say We have two sites, A and B, from which inference requests are being made. Site A has 1000 inference requests, while site B has 10. Currently, the 10 requests from site B are queued behind the 1000 requests from site A, causing unnecessary delays.
What I aim to achieve is to enable these 10 requests from site B to proceed independently of the requests from site A, without having to wait for all 1000 requests to complete.
So i was thinking like the number of queues need to be equal number if sites we have. I can be completely wrong.