Hi Too often something happens and flow freezes influentiall

Question

Hi Too often something happens and flow freezes influentially Flow should finish in few seconds sometimes it work but sometimes it freezes forever It is impossible to cancel it too How to debug such problem 10 of flow runs have such behaviour and it happens only if server side run in docker

Jenny · Answer

Hi Ruslan Double checking that this is happening on Prefect Cloud Looks like you have an automation to cancel the run Does the flow run enter a cancelling state if you use the cancel button in the UI

Ruslan · Answer

it would not stop if automation exucutes or if I cancell it manually It will gon in influent cancelling state the problem is as I understand that agent went away for some reason but it work well and other flows working on that agent and ui influently waiting for this agent

Jenny · Answer

Hmm yes it looks as there s a problem for cloud to communicate with your infrastructure The Cancel button and cancel flow run automation are best effort attempts to stop all flows but sometimes they can not reach the flow infrastructure and things get stuck Are you concerned that you flow is still running can you stop that externally Or do you want them marked as cancelled in Cloud if so you can use the set state drop down and mark them as cancelled

Ruslan · Answer

yes usually a set manually status cancelled How to stop flow externally anyway how to debug this problem I don t understand what happening infrastructure work perfectly on my side more than that the same agent working well with another flows while one of them get stucked

Jenny · Answer

Can you share the agent logs And perhaps flow logs too

Ruslan · Answer

``` 2021 08 05 13 52 54 909 DEBUG External01 | Heartbeat succesful Sleeping for 60 0 seconds 2021 08 05 13 53 03 038 DEBUG External01 | Querying for ready flow runs 2021 08 05 13 53 03 213 DEBUG External01 | No ready flow runs found 2021 08 05 13 53 03 214 DEBUG External01 | Sleeping flow run poller for 10 0 seconds 2021 08 05 13 53 13 214 DEBUG External01 | Querying for ready flow runs 2021 08 05 13 53 13 368 DEBUG External01 | No ready flow runs found 2021 08 05 13 53 13 368 DEBUG External01 | Sleeping flow run poller for 10 0 seconds 2021 08 05 13 53 23 369 DEBUG External01 | Querying for ready flow runs 2021 08 05 13 53 23 526 DEBUG External01 | No ready flow runs found 2021 08 05 13 53 23 526 DEBUG External01 | Sleeping flow run poller for 10 0 seconds 2021 08 05 13 53 33 527 DEBUG External01 | Querying for ready flow runs 2021 08 05 13 53 33 713 DEBUG External01 | No ready flow runs found 2021 08 05 13 53 33 714 DEBUG External01 | Sleeping flow run poller for 10 0 seconds 2021 08 05 13 53 43 714 DEBUG External01 | Querying for ready flow runs 2021 08 05 13 53 43 877 DEBUG External01 | No ready flow runs found 2021 08 05 13 53 43 877 DEBUG External01 | Sleeping flow run poller for 10 0 seconds 2021 08 05 13 53 53 877 DEBUG External01 | Querying for ready flow runs 2021 08 05 13 53 54 055 DEBUG External01 | No ready flow runs found 2021 08 05 13 53 54 055 DEBUG External01 | Sleeping flow run poller for 10 0 seconds 2021 08 05 13 53 54 959 DEBUG External01 | Sending agent heartbeat 2021 08 05 13 53 54 959 DEBUG External01 | Heartbeat succesful Sleeping for 60 0 seconds ``` I don t see anything in agent logs only this in a loop

Ruslan · Answer

this is flow logs I made detailed log in python source so you can see that every ~7sec 10000 rows converted but in some moment it freezes forever

Ruslan · Answer

source working 100 on same machine I tried local agent then I create docker with local agent on the same machine and it have such problem

Jenny · Answer

Ok Thanks for those I ll see if the wider team have any ideas

Jenny · Answer

Hi Ruslan A few follow up questions suggestions Are you using docker here If so what do the container logs show And which executor type are you using If LocalDaskExecutor a lot of people have success using processes over threads

Ruslan · Answer

Hi Jenny I ve attached docker container logs I don t see something interesting there Yes I use local agent inside docker Executer is LocalDaskExecutor scheduler= threads num workers=1

Kevin Kho · Answer

Have you tried using processes Ruslan

Ruslan · Answer

Hi Kevin may you clarify please what does it mean to use processes

Ruslan · Answer

ah clear sorry let me try

Ruslan · Answer

I guess it didn t help more than that now flow freezes in 100 cases

Kevin Kho · Answer

Oh just saw this This is related to your recent post right

Ruslan · Answer

< Jenny> < Kevin Kho> thank you scheduler= processes work stable second problem on was on my side because of memory size

Kevin Kho · Answer

Glad to hear