Hi We are intermittently getting this error in some of our p

Question

Hi We are intermittently getting this error in some of our prefect flows running on 2 8 4 and tried increasing resources as well but nothing works If someone has faced this earlier please help Setup is running on kubernetes ``` AssertionError Traceback most recent call last File usr lib python3 8 asyncio runners py line 44 in run return loop run until complete main File usr lib python3 8 asyncio base events py line 603 in run until complete self run forever File usr lib python3 8 asyncio base events py line 570 in run forever self run once File usr lib python3 8 asyncio base events py line 1823 in run once event list = self selector select timeout File usr lib python3 8 selectors py line 468 in select fd event list = self selector poll timeout max ev File usr local lib python3 8 dist packages prefect engine py line 1670 in cancel flow run raise TerminationSignal signal=signal SIGTERM prefect exceptions TerminationSignal During handling of the above exception another exception occurred Traceback most recent call last File usr local lib python3 8 dist packages prefect engine py line 1533 in orchestrate task run result = await run sync task fn args kwargs File usr local lib python3 8 dist packages prefect utilities asyncutils py line 165 in run sync in interruptible worker thread assert result is not NotSet AssertionError Finished in state Failed Task run encountered an exception Traceback most recent call last n File usr lib python3 8 asyncio runners py line 44 in run n return loop run until complete main n File usr lib python3 8 asyncio base events py line 603 in run until complete n self run forever n File usr lib python3 8 asyncio base events py line 570 in run forever n self run once n File usr lib python3 8 asyncio base events py line 1823 in run once n event list = self selector select timeout n File usr lib python3 8 selectors py line 468 in select n fd event list = self selector poll timeout max ev n File usr local lib python3 8 dist packages prefect engine py line 1670 in cancel flow run n raise TerminationSignal signal=signal SIGTERM nprefect exceptions TerminationSignal n nDuring handling of the above exception another exception occurred n nAssertionError n ```

Ankit · Answer

Hi can someone please help with this Thanks

jpuris · Answer

~Your pod is getting killed by k8s Please see kubernetes pod metadata for the reason this is happening ~ ~edit most likely out of memory or getting evicted~ shrug

jpuris · Answer

Actually looking at this again I do not think this is k8s doing this at all it seems prefect is doing graceful shutdown SIGTERM due to an exception in cancelling the flow run

jpuris · Answer

I suspect these are not full logs It would be interesting to see why the cancellation was emitted in the first place

jpuris · Answer

Anyhow from what I can tell in the logs is that ```prefect exceptions TerminationSignal``` a SIGTERM was received possibly from k8s killing the pod please do check the pod metadata i e is the status `terminated` if so what is the reason or `failed` When the graceful shutdown was init the code threw another exception ```AssertionError``` I do not know what this is

Ankit · Answer

Hi < jpuris> this is the full logs I did check all pod metrics and both memory and cpu are over provisioned by 5x of what the flow actually needs And only 2 out of 40 flows are facing this issue intermittently At times flow status ends in Success and goes unnoticed that it actually failed

jpuris · Answer

What does kubernetes tell you about this pod Was it terminated or did it exit with bad status i e runtime error

Ankit · Answer

pod was terminated normally all pod metrics look normal as well

redsquare · Answer

Hey < Ankit> I am seeing the exact same intermittent issues in k8s have you resolved

Ankit · Answer

Hey < redsquare> no it s still happening quite frequently

jpuris · Answer

Kubernetes is sending a SIGTERM to the pod which is a graceful shutdown When such signal has been received prefect emits following exception <https docs prefect io 2 11 2 api ref prefect exceptions prefect exceptions TerminationSignal|TerminationSignal> Unless I m terribly wrong here the answer is in kubernetes You d need to find out why k8s scales down the pod

redsquare · Answer

yeah I cannot see any reason all pod stats look healthy in datadog

jpuris · Answer

Unfortunately I m not familiar with datadog shrug There is no termination reason for the pod You d be able to see this in kubectl Alternative theory is that the flow was cancelled it may be already fixed gt btw how should I distinguish a `spot instance` shutdown vs user initiated `cancel` in k8s Assume I have the ASG with spot instances when the nodes are being released I would get `SIGTERM` in the running pods too Source <https prefect community slack com archives C048SVCEFF0 p1690709891457919> If you can reproduce this reliably a MRE would be super helpful confused

redsquare · Answer

I ll check the pod termination reason the next time it happens I ttl them pretty quickly afterwards

redsquare · Answer

I can reproduce this daily btw long running flow 3k tasks randomly fails between 10 mins gt 2 hours and everything in between

redsquare · Answer

image png

redsquare · Answer

no other pods on the node or in the cluster get terminated around the same time

Ankit · Answer

I have tried change node types as well there is no pattern

redsquare · Answer

we are on eks

redsquare · Answer

has to be something with the prefect event loop

redsquare · Answer

< jpuris> I may have something just got another flow to fail but the flow amp agent logs are interesting the flow appears to start again notice the downloading flow code

redsquare · Answer

the agent meanwhile reported 2023 08 04T08 15 12 362509340Z 08 13 38 857 | INFO | Flow run meek limpet Created task run update rooftop metrics 3407 for task update rooftop metrics 2023 08 04T08 15 12 362511460Z 08 13 38 859 | INFO | Flow run meek limpet Executing update rooftop metrics 3407 immediately 2023 08 04T08 15 12 362513900Z rpc error code = NotFound desc = an error occurred when try to find container f782a10e7abe00b4484840abd243bd0eabdcfc5dcf0e7a242812035ffc31c4e1 not found Fri Aug 4 2023 9 15 12 am +pip install s3fs==2023 1 0 pydantic==1 10 6 datadog python dotenv r7insight python python dotenv prefect kv r7insight python datadog psycopg2 binary clickhouse connect

redsquare · Answer

I assume the rpc error is k8s logging but cant be 100 I cannot find that container id anywhere

redsquare · Answer

image png

redsquare · Answer

I have narrowed this down I have created a dedicated agent in a container in the same pod for this flow and it has run twice with no issues

redsquare · Answer

< Ankit> maybe something for you to try

Ankit · Answer

Thanks < redsquare> I will also try this out

Yuriy Trofimentsev · Answer

Hi My flows also experience this issue Have you figured out the problem Today I updated the version of Prefect to 2 11 3 but the problem is still there

redsquare · Answer

nope still trying to find the issue

Yuriy Trofimentsev · Answer

I tried to reduce the number of concurrency limit in work pool and my flows finished successfully but really long ofk so most likely the problem indeed is connected with concurrency issues

redsquare · Answer

I am using the sequential runner

Yuriy Trofimentsev · Answer

< redsquare> how do you run your flow Do you have subflows or just sequential tasks inside

redsquare · Answer

purely sequential tasks 4k of them that just do a few http calls

Yuriy Trofimentsev · Answer

And on visualization panel you see that they are running one by one

redsquare · Answer

that doesnt draw correctly for me the logs suggest they

redsquare · Answer

found one that does

Yuriy Trofimentsev · Answer

Do you use the results from the previous task in the next one

redsquare · Answer

nope

redsquare · Answer

basically iterate a dataframe and call an api for each row

redsquare · Answer

as simple as it gets

redsquare · Answer

or so you would hope

Yuriy Trofimentsev · Answer

Got it thanks for answers I will be back if I find something

Yuriy Trofimentsev · Answer

This error happens to me from time to time too I have a flow that launches many other subflows Some of these subflows are falling It is noteworthy that they always crash if run together but if run separately no errors occur Most often the flow with this code falls I ll share the code in the thread maybe it will help in finding the error ``` task def select sql path str params BigQueryParamsMixin gt pd DataFrame query = render sql path params=params render params job config = bq QueryJobConfig query parameters=params bq params return bq client query query location= EU job config=job config to dataframe def split params BigQueryParamsMixin chunks size int = 150 000 input data = getattr params input data pd DataFrame chunks number = len input data index chunks size if chunks number gt = 1 for chunk in np array split input data chunks number params copy = deepcopy params the task which saves the result in GCS input data path = save result chunk setattr params copy input data path input data path yield params copy else yield params flow task runner=SequentialTaskRunner def select metric sql str params BigQueryParamsMixin from file bool = True gt pd DataFrame df = pd DataFrame for params in split params BigQuery has limits on the size of a query Therefore it is worth splitting the input into chunks if it s needed df 2 = select sql params df = pd concat df df 2 return df``` And also traceback which is displayed when the error occurs ```Crash details Traceback most recent call last File usr local lib python3 11 site packages prefect engine py line 1884 in capture sigterm yield File usr local lib python3 11 site packages prefect internal concurrency api py line 243 in wait for call in loop thread return call result ^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect internal concurrency calls py line 283 in result return self future result timeout=timeout ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect internal concurrency calls py line 169 in result return self get result ^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 concurrent futures base py line 401 in get result raise self exception File usr local lib python3 11 site packages prefect internal concurrency calls py line 346 in run async result = await coro ^^^^^^^^^^ File usr local lib python3 11 site packages prefect client utilities py line 51 in with injected client return await fn args kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect engine py line 681 in create and begin subflow run terminal state = await orchestrate flow run ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect engine py line 830 in orchestrate flow run result = await flow call aresult ^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect internal concurrency calls py line 292 in aresult return await asyncio wrap future self future ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect internal concurrency calls py line 316 in run sync result = self fn self args self kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File opt trebufect flows select metric py line 53 in select metric df 2 = select sql params ^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect tasks py line 730 in submit return enter task run engine ^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect engine py line 1145 in enter task run engine return from sync wait for call in loop thread begin run ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages prefect internal concurrency api py line 242 in wait for call in loop thread waiter wait File usr local lib python3 11 site packages prefect internal concurrency waiters py line 152 in wait self handle waiting callbacks File usr local lib python3 11 site packages prefect internal concurrency waiters py line 119 in handle waiting callbacks callback Call = self queue get ^^^^^^^^^^^^^^^^^ File usr local lib python3 11 queue py line 171 in get self not empty wait File usr local lib python3 11 threading py line 320 in wait waiter acquire File usr local lib python3 11 site packages prefect engine py line 1874 in cancel flow run raise TerminationSignal signal=signal SIGTERM prefect exceptions TerminationSignal``` Prefect Version `2 11 3` Python Version `3 11` I have one default agent Each subflow runs its own Kubernetes Job

Yuriy Trofimentsev · Answer

Could someone take a look and help us please pray

redsquare · Answer

< Yuriy Trofimentsev> oom

redsquare · Answer

< Yuriy Trofimentsev> I have fixed my issue btw I changed my prefect tasks that query the db to just normal functions

Yuriy Trofimentsev · Answer

< redsquare> gt oom Not sure I monitored the memory usage of pods and it s fine gt I changed my prefect tasks that query the db to just normal functions Yes I m sure that it can fix the problem if rewrite the code from tasks to basic functions However I d like to use Prefect with tasks slightly smiling face

redsquare · Answer

yeah I just needed mine to run to completion

redsquare · Answer

wasted 4 days trying to work it out

Ankit · Answer

I instead used Sequential Task Runner it still fails but not that frequently

redsquare · Answer

yeah I already have that

redsquare · Answer

< Yuriy Trofimentsev> are yours long lasting jobs

Yuriy Trofimentsev · Answer

Generally 1 40 mins it depends of data size However the flow can fall both after 1 5 minutes and after 30 Also the size of the input data does not affect whether the flow will fall or not Usually from a set of 12 flows it falls about 3 4 but unsuccessful flows are usually different

redsquare · Answer

yeah there is nothing to go off due to the randomness

Yuriy Trofimentsev · Answer

In my case the problem was in cluster autoscaller I set the next annotation to prevent this behavior ` false ` So piece from my deployment ``` job apiVersion batch v1 kind Job metadata labels spec template metadata annotations false ``` Not sure that it can solve your cases but anyway

redsquare · Answer

interesting but I would expect this to show in the logs somewhere

Tanishq Hooda · Answer

Even I m facing this issue Were you able to resolve it < Yuriy Trofimentsev> < redsquare> < Ankit> Even my tasks are long 2 3 hours Prefect Version 2 10 13 Apart from Termination error I also have a lot of these in my logs Prefect community says it can be ignored wondering if they are related Anyway I ve attached the complete traceback ``` usr local lib python3 9 runpy py 127 RuntimeWarning prefect engine found in sys modules after import of package prefect but prior to execution of prefect engine this may result in unpredictable behaviour warn RuntimeWarning msg ```

Ankit · Answer

No still facing this We are moving out of prefect for critical flows and exploring other options