Hi! Too often something happens and flow freezes influentially. Flow should finish in few seconds, sometimes it work but sometimes it freezes forever. It is impossible to cancel it too. How to debug such problem? 10% of flow runs have such behaviour and it happens only if server side run in docker

Hi Ruslan! Double checking that this is happening on Prefect Cloud? Looks like you have an automation to cancel the run? Does the flow run enter a cancelling state if you use the cancel button in the UI?

it would not stop if automation exucutes or if I cancell it manually. It will gon in influent “cancelling” state. the problem is as I understand that agent went away for some reason (but it work well and other flows working on that agent) and ui influently waiting for this agent

Hmm yes it looks as there's a problem for cloud to communicate with your infrastructure. The "Cancel" button (and cancel flow run automation) are best effort attempts to stop all flows but sometimes they can not reach the flow infrastructure and things get stuck. Are you concerned that you flow is still running - can you stop that externally? Or do you want them marked as cancelled in Cloud - if so you can use the set state drop down and mark them as cancelled.

yes usually a set manually status “cancelled”. How to stop flow externally? anyway how to debug this problem, I don’t understand what happening, infrastructure work perfectly on my side more than that the same agent working well with another flows while one of them get stucked

Can you share the agent logs? And perhaps flow logs too?

[2021-08-05 13:52:54,909] DEBUG - External01 | Heartbeat succesful! Sleeping for 60.0 seconds...
[2021-08-05 13:53:03,038] DEBUG - External01 | Querying for ready flow runs...
[2021-08-05 13:53:03,213] DEBUG - External01 | No ready flow runs found.
[2021-08-05 13:53:03,214] DEBUG - External01 | Sleeping flow run poller for 10.0 seconds...
[2021-08-05 13:53:13,214] DEBUG - External01 | Querying for ready flow runs...
[2021-08-05 13:53:13,368] DEBUG - External01 | No ready flow runs found.
[2021-08-05 13:53:13,368] DEBUG - External01 | Sleeping flow run poller for 10.0 seconds...
[2021-08-05 13:53:23,369] DEBUG - External01 | Querying for ready flow runs...
[2021-08-05 13:53:23,526] DEBUG - External01 | No ready flow runs found.
[2021-08-05 13:53:23,526] DEBUG - External01 | Sleeping flow run poller for 10.0 seconds...
[2021-08-05 13:53:33,527] DEBUG - External01 | Querying for ready flow runs...
[2021-08-05 13:53:33,713] DEBUG - External01 | No ready flow runs found.
[2021-08-05 13:53:33,714] DEBUG - External01 | Sleeping flow run poller for 10.0 seconds...
[2021-08-05 13:53:43,714] DEBUG - External01 | Querying for ready flow runs...
[2021-08-05 13:53:43,877] DEBUG - External01 | No ready flow runs found.
[2021-08-05 13:53:43,877] DEBUG - External01 | Sleeping flow run poller for 10.0 seconds...
[2021-08-05 13:53:53,877] DEBUG - External01 | Querying for ready flow runs...
[2021-08-05 13:53:54,055] DEBUG - External01 | No ready flow runs found.
[2021-08-05 13:53:54,055] DEBUG - External01 | Sleeping flow run poller for 10.0 seconds...
[2021-08-05 13:53:54,959] DEBUG - External01 | Sending agent heartbeat...
[2021-08-05 13:53:54,959] DEBUG - External01 | Heartbeat succesful! Sleeping for 60.0 seconds...

I don’t see anything in agent logs, only this in a loop

Ok. Thanks for those. I'll see if the wider team have any ideas.

Hi Jenny! I’ve attached docker container logs.. I don’t see something interesting there… Yes I use local agent inside docker. Executer is LocalDaskExecutor(scheduler=“threads”, num_workers=1)

Have you tried using processes Ruslan?

Hi Kevin, may you clarify please what does it mean to use processes?

Oh just saw this. This is related to your recent post right?

@Jenny @Kevin Kho thank you! scheduler=“processes” work stable! second problem on was on my side because of memory size

Glad to hear