https://prefect.io logo
r

Ruslan

08/05/2021, 5:27 AM
Hi! Too often something happens and flow freezes influentially. Flow should finish in few seconds, sometimes it work but sometimes it freezes forever. It is impossible to cancel it too. How to debug such problem? 10% of flow runs have such behaviour and it happens only if server side run in docker
j

Jenny

08/05/2021, 1:26 PM
Hi Ruslan! Double checking that this is happening on Prefect Cloud? Looks like you have an automation to cancel the run? Does the flow run enter a cancelling state if you use the cancel button in the UI?
r

Ruslan

08/05/2021, 3:03 PM
it would not stop if automation exucutes or if I cancell it manually. It will gon in influent “cancelling” state. the problem is as I understand that agent went away for some reason (but it work well and other flows working on that agent) and ui influently waiting for this agent
j

Jenny

08/05/2021, 4:54 PM
Hmm yes it looks as there's a problem for cloud to communicate with your infrastructure. The "Cancel" button (and cancel flow run automation) are best effort attempts to stop all flows but sometimes they can not reach the flow infrastructure and things get stuck. Are you concerned that you flow is still running - can you stop that externally? Or do you want them marked as cancelled in Cloud - if so you can use the set state drop down and mark them as cancelled.
r

Ruslan

08/06/2021, 4:42 PM
yes usually a set manually status “cancelled”. How to stop flow externally? anyway how to debug this problem, I don’t understand what happening, infrastructure work perfectly on my side more than that the same agent working well with another flows while one of them get stucked
j

Jenny

08/06/2021, 5:07 PM
Can you share the agent logs? And perhaps flow logs too?
r

Ruslan

08/06/2021, 6:50 PM
Copy code
[2021-08-05 13:52:54,909] DEBUG - External01 | Heartbeat succesful! Sleeping for 60.0 seconds...
[2021-08-05 13:53:03,038] DEBUG - External01 | Querying for ready flow runs...
[2021-08-05 13:53:03,213] DEBUG - External01 | No ready flow runs found.
[2021-08-05 13:53:03,214] DEBUG - External01 | Sleeping flow run poller for 10.0 seconds...
[2021-08-05 13:53:13,214] DEBUG - External01 | Querying for ready flow runs...
[2021-08-05 13:53:13,368] DEBUG - External01 | No ready flow runs found.
[2021-08-05 13:53:13,368] DEBUG - External01 | Sleeping flow run poller for 10.0 seconds...
[2021-08-05 13:53:23,369] DEBUG - External01 | Querying for ready flow runs...
[2021-08-05 13:53:23,526] DEBUG - External01 | No ready flow runs found.
[2021-08-05 13:53:23,526] DEBUG - External01 | Sleeping flow run poller for 10.0 seconds...
[2021-08-05 13:53:33,527] DEBUG - External01 | Querying for ready flow runs...
[2021-08-05 13:53:33,713] DEBUG - External01 | No ready flow runs found.
[2021-08-05 13:53:33,714] DEBUG - External01 | Sleeping flow run poller for 10.0 seconds...
[2021-08-05 13:53:43,714] DEBUG - External01 | Querying for ready flow runs...
[2021-08-05 13:53:43,877] DEBUG - External01 | No ready flow runs found.
[2021-08-05 13:53:43,877] DEBUG - External01 | Sleeping flow run poller for 10.0 seconds...
[2021-08-05 13:53:53,877] DEBUG - External01 | Querying for ready flow runs...
[2021-08-05 13:53:54,055] DEBUG - External01 | No ready flow runs found.
[2021-08-05 13:53:54,055] DEBUG - External01 | Sleeping flow run poller for 10.0 seconds...
[2021-08-05 13:53:54,959] DEBUG - External01 | Sending agent heartbeat...
[2021-08-05 13:53:54,959] DEBUG - External01 | Heartbeat succesful! Sleeping for 60.0 seconds...
I don’t see anything in agent logs, only this in a loop
this is flow logs I made detailed log in python source, so you can see that every ~7sec 10000 rows converted, but in some moment it freezes forever
source working 100% on same machine, I tried local agent. then I create docker with local agent on the same machine and it have such problem
j

Jenny

08/06/2021, 7:50 PM
Ok. Thanks for those. I'll see if the wider team have any ideas.
Hi Ruslan. A few follow up questions/suggestions: Are you using docker here? If so, what do the container logs show? And which executor type are you using? If LocalDaskExecutor, a lot of people have success using processes over threads.
r

Ruslan

08/07/2021, 10:03 AM
Hi Jenny! I’ve attached docker container logs.. I don’t see something interesting there… Yes I use local agent inside docker. Executer is LocalDaskExecutor(scheduler=“threads”, num_workers=1)
k

Kevin Kho

08/08/2021, 2:58 PM
Have you tried using processes Ruslan?
r

Ruslan

08/09/2021, 7:52 AM
Hi Kevin, may you clarify please what does it mean to use processes?
ah clear sorry, let me try
I guess it didn’t help… more than that now flow freezes in 100% cases
k

Kevin Kho

08/10/2021, 4:02 PM
Oh just saw this. This is related to your recent post right?
r

Ruslan

08/13/2021, 6:39 PM
@Jenny @Kevin Kho thank you! scheduler=“processes” work stable! second problem on was on my side because of memory size
👍 2
k

Kevin Kho

08/13/2021, 6:39 PM
Glad to hear
upvote 1
3 Views