https://prefect.io logo
Title
r

Richard Alexander

12/19/2022, 3:28 PM
I have a question about the timing of Agent polling in Prefect 2. I believe in Prefect 1, agents polled for new flows every 10 seconds. This seems not to be the case in my 2.6.4 install. This is a fairly new install with very little going on. But I have a work queue that is not starting flows until they are 5-10 min late. (see attached screenshot) I have a concurency limit of 3, and I'm populating the queue by immediately scheduling the task via a script which calls
run_deployment
with
timeout=0
. I can submit many flows to an empty queue, but none of the flows start until they are very late as seen in the screenshot. Any idea what is going on here?
a

Anna Geller

12/19/2022, 3:41 PM
this is adjustable with the setting PREFECT_AGENT_QUERY_INTERVAL='10.0' this may also help https://discourse.prefect.io/t/my-flow-run-in-prefect-2-is-stuck-in-a-pending-state-what-can-i-do/2012
r

Richard Alexander

12/19/2022, 4:12 PM
Thanks for that info @Anna Geller, but I don't think it helps my situation. I've done some more digging and the agent query interval has never been changed (I don't see it when I run
prefect config view
). I watched
htop
for a while and it seems like both my queues (I only have 2) are polling at least every 10 seconds. So that being the case, I can't understand why the jobs aren't being picked up. I have 4 CPU cores and they are certainly not busy. To be clear, the other queue does not have anything going on that could block resources. It has a single flow scheduled to run every 5 minutes, and that flow typically returns in under 2 seconds. What could be preventing my immediately-scheduled flows from being picked up by an agent for so many minutes when we seem to be polling so often? Do you have any other suggestions of how I might troubleshoot this situation?
I'm attaching some additional results from my latest test run. These results highlight the painfulness of this bug. My flows that take under a minute to complete are being delayed for 30-45 minutes before they start. (see screenshot)
a

Anna Geller

12/19/2022, 5:30 PM
usually pending runs indicate issue in the execution layer, but if you are confident enough that this is a Prefect bug, it would be great to open a GitHub issue with a minimal reproducible example and we could investigate. Add as much detail as possible to reproduce incl. details on your version, infra and the way you set that up. :gratitude-thank-you:
b

Ben Muller

12/19/2022, 6:27 PM
@Richard Alexander I had some similar issues previously. I'd upgrade to the most recent version of prefect. All the issues are fixed now.
:gratitude-thank-you: 1
Oh and also turning off HTTP2 on the agent. I can't remember what the environment variable was I needed to set (on my phone rn)
🙌 1
👍 1
:gratitude-thank-you: 1