I have a question about the timing of Agent polling in Prefect 2. I believe in Prefect 1, agents polled for new flows every 10 seconds. This seems not to be the case in my 2.6.4 install. This is a fairly new install with very little going on. But I have a work queue that is not starting flows until they are 5-10 min late. (see attached screenshot) I have a concurency limit of 3, and I'm populating the queue by immediately scheduling the task via a script which calls

run_deployment

with

timeout=0

. I can submit many flows to an empty queue, but none of the flows start until they are very late as seen in the screenshot. Any idea what is going on here?

this is adjustable with the setting PREFECT_AGENT_QUERY_INTERVAL='10.0' this may also help https://discourse.prefect.io/t/my-flow-run-in-prefect-2-is-stuck-in-a-pending-state-what-can-i-do/2012

Thanks for that info @Anna Geller, but I don't think it helps my situation. I've done some more digging and the agent query interval has never been changed (I don't see it when I run

prefect config view

). I watched

htop

for a while and it seems like both my queues (I only have 2) are polling at least every 10 seconds. So that being the case, I can't understand why the jobs aren't being picked up. I have 4 CPU cores and they are certainly not busy. To be clear, the other queue does not have anything going on that could block resources. It has a single flow scheduled to run every 5 minutes, and that flow typically returns in under 2 seconds. What could be preventing my immediately-scheduled flows from being picked up by an agent for so many minutes when we seem to be polling so often? Do you have any other suggestions of how I might troubleshoot this situation?

usually pending runs indicate issue in the execution layer, but if you are confident enough that this is a Prefect bug, it would be great to open a GitHub issue with a minimal reproducible example and we could investigate. Add as much detail as possible to reproduce incl. details on your version, infra and the way you set that up. :gratitude-thank-you:

@Richard Alexander I had some similar issues previously. I'd upgrade to the most recent version of prefect. All the issues are fixed now.