Dan Wise

11/09/2022, 11:48 AM
Hi does anyone know in Prefect 2 why there would be a 30 second delay between the agent picking up a flow run and starting to execute it? Many thanks.


11/09/2022, 12:11 PM
I have my flows stored in GitHub storage and I always assume this is the time it takes to clone everything and setup the infrastructure... But this is only an assumption and I would be happy to learn more about this!

Thomas Pedersen

11/09/2022, 12:15 PM

Dan Wise

11/09/2022, 12:17 PM
Well I am seeing that it checks for runs every few seconds but the delay is after it picks up a run but before it starts executing. Seems it takes a long time to create a subprocess on the box the agent is on
In the subprocess it runs python -m prefect.engine

Pierre Monico

11/09/2022, 1:07 PM
In general I also noticed there is a delay in running flows locally since we migrated to prefect 2
Probably linked to all the background stuff it needs to spin up in order to track the runs

Dan Wise

11/09/2022, 1:12 PM
Hi, I put in debug statements to the agent and it's the get_directory call on the storage which takes the time. Seems the subprocess spawned by the agent copies everything into a directory under the user's temp
Not sure why that is necessary really as I am using local storage so it does not need to copy from one local location to another

Christopher Boyd

11/09/2022, 1:48 PM
permissions might not be the same for the directories locally, since where your code and execution is stored is presumably owned not by the prefect account
you might have read permissions for the user, but not execute permissions

Dan Wise

11/09/2022, 2:40 PM
Actually I found a way round the copy. I created a Process block with the working directory the same as the one I use for the deployment. No copy is therefore required and it works really nicely.
🎉 1