Alix Cook
02/02/2023, 12:00 AMPending
state. We're running with Process
infrastructure and github storage. We're running the agent in the container, and the agent is pid 1, and I know that the agent process that picked up the flow is still alive (we have the process ping for all the flow runs its submitted).
because we run many things with strict concurrency, this issue is blocking our flows from running unless we kill the agent or cancel the flow. any thoughts?Christopher Boyd
02/02/2023, 2:40 PMAlix Cook
02/02/2023, 2:51 PMprefect.engine Engine execution of flow run 'c2dbdc39-7f36-4e31-b3e3-f7275dab1257' aborted by orchestrator: Error validating state: DBAPIError("(sqlalchemy.dialects.postgresql.asyncpg.Error) <class 'asyncpg.exceptions.QueryCanceledError'>: canceling statement due to statement timeout")
around when it looks like the flow "completed" then continued to report its state as runningChristopher Boyd
02/03/2023, 3:38 PMAlix Cook
02/03/2023, 3:46 PMChristopher Boyd
02/03/2023, 3:49 PMProcess
? When did this behavior start?
What version of prefect are you using, and how is the deployment registered?
Once the flows run, do they actually run successfully?
Are you able to enable PREFECT_LOGGING_LEVEL=DEBUG on the agent to see as its polling for scheduled flow runs?Alix Cook
02/03/2023, 4:06 PMEmilie Hester
02/03/2023, 6:35 PMAlix Cook
02/03/2023, 10:45 PMFeb 3 15:02:10 d559943086fb production-api DEBUG prefect.task_runner.concurrent Shutting down task runner...
Feb 3 15:02:10 d559943086fb production-api INFO prefect.engine Engine execution of flow run 'ccd83d61-086d-4dfc-8aaa-ce54e6459568' aborted by orchestrator: Error validating state: DBAPIError("(sqlalchemy.dialects.postgresql.asyncpg.Error) <class 'asyncpg.exceptions.QueryCanceledError'>: canceling statement due to statement timeout")
but i also don't really see that flow run mentioned after that at allChristopher Boyd
02/03/2023, 10:50 PMAlix Cook
02/03/2023, 10:55 PMChristopher Boyd
02/13/2023, 4:34 PMAlix Cook
02/13/2023, 5:25 PMPending
flows issue i initially reported, I'll try to get more data on thatPending
issue? we are already accessing the flow run object so can log info about it. Debug logs aren't really showing anything that seems relatedChristopher Boyd
02/14/2023, 1:30 AMAlix Cook
02/14/2023, 3:30 PMCheck if there is more than one agent polling for runs from the same work queueyes, we frequently do this because we have to run things in a local process right now, and we needed to scale that horizontally. I was told this was a fine use case, is that no longer true? I'll try turning down the polling interval.
Emilie Hester
02/14/2023, 3:57 PMChristopher Boyd
02/14/2023, 3:59 PMAlix Cook
02/14/2023, 4:41 PM