Hi everyone / @Marvin I use a docker worker pool with fly machines as the workers, but it seems that work is not being nicely distributed between them. The workers are booted up with the

--limit 3

but i find that even though there's more than enough work - those workers are not being fully utilised. Has anyone ran into something like this and or has a cool solution, is there something I'm maybe missing? I have not applied any concurrency limit's other than on some deployments, but even then there are enough deployments without concurrency limits to utilise both of my worker machines

It seems there was a slight glitch with my search functions—most likely my enthusiasm for finding an issue outran the system's capacity to handle it. Let me try that again with a bit more clockwork precision. I'll look into what might be causing the imbalance in worker utilization. Give me a moment! I've had a look through the available documentation on Prefect's 2.x and 3.x versions, as well as GitHub, to see if there's any noted issue or guidance regarding your situation with worker load distribution. From what I found in the documentation: 1. Workers & Work Pools: - Workers in Prefect are responsible for polling work pools and executing scheduled runs. Each worker is associated with a specific type that matches the execution environment they interact with. It's essential to ensure that the configuration you're using for workers aligns with this setup. You can read more about this in the Prefect 2.x Work Pools documentation and the 3.x Workers documentation. 2. Load Balancing & Work Queues: - Work queues play a role in determining how work is distributed among workers. The setup of these queues, including their priorities and paused status, can affect worker utilization. It's important to make sure all configurations are correct and that none of your work queues are unintentionally paused or prioritized incorrectly. 3. Dynamic Infrastructure: - It's also worth considering if your infrastructure configuration allows for dynamic provisioning. This could be a reason behind any perceived underutilization. Prefect supports both pull and push work pools, which could influence how work is assigned to workers. However, I didn't find any specific open issues on GitHub that directly addressed the problem you're experiencing with workload distribution in a Docker worker pool setup. It might be a good idea to double-check your work queue configurations and worker settings or consider reaching out to the Prefect community for more insight if these suggestions don't help. Isn't it just typical though? You try to balance workload, and things just don't cooperate. It's almost like dealing with Marvin's mood swings, but hopefully, a bit less morose.