Trying my luck again, hi everyone, I am recently facing a strange problem. I have a flow (which is a subflow of another one) that runs multiple tasks. Once in a while the tasks do not even start (get stuck on pending state) and the logs of each tasks says "Crash detected! Execution was cancelled by the runtime environment." The main flow is stuck on running state and the log says: " Flow run '...' - Crash detected! Request to http://.../api/task_runs faield: PoolTimeout: . The main problem is that the tasks & subflow get stuck and even a timeout (timeout_seconds param) does not help to release the freeze... (The flow is still in running state...) I am running the flows as a k8s job on openshift and all the flows and tasks are async. Is it a known problem? Is calling an async function (as a subflow or task) could be problematic?😵‍💫

Just to clarify: do the processes themselves freeze where they are running or do they only show as running in the prefect UI?

@Dev Dabke I checked again and it seems that the processes enters an infinity loop with an asyncio exception:

asyncio - Exception in callback SubprocessStreamProtocol.pipe_data_recieved(2,...)
Traceback (most recent call last):
File "/usr/local/lib/pyhton3.8/asyncio/events.py", line81 in _run
self._context.run(self._callback, *self.args)
File "/usr/local/lib/pyhton3.8/asyncio/subprocess.py", line 73, in pipe_data_recieved
reader.feed_data(data)
File "/usr/local/lib/pyhton3.8/asyncio/streams.py", line 472, in feed_data
assert not self._eof, 'feed_data after feed_eof'
AssertionError: feed_data after feed_eof

It's important to say that I'm currently running in Debug Mode for investigating the problem, so this new data might be a cause of the debugging process 🤔

One q: are you main-guarding your flow?

if __name__ == "__main__":
  flow_fn()

I am not

Okay, I would try that. async is super broken without the main guard.

Yeah it's a great idea

How are you launching the flow? From the UI?

Yes

Okay, then I suspect it's not the main guarding that's at issue.

I didn't tried to call it locally. This problem happens only when my flow runs for a long time (a few hours). And in my flow I await the subflow and await a gather of all the tasks in each subflow

Ah gotcha, okay so it seems like there's two issues: 1. The underlying error: it seems like an I/O issues (is the network getting interrupted? is the computer going to sleep?) 2. The fact that the flow isn't being marked as crashed. Is it possible that you're catching the error with

except

and not re-raising it? Without seeing the code, it's a bit hard to guess as to why the error is not causing the processes to crash.

I don't think the network is the problem, on my checks it looked fine and nothing seemed not normal. About the crashed/stuck problem. I tried to make the main flow to crash when the subflow is stuck, it didn't work 100% of the times. So to sum it up: Main flow - sometimes crash and sometimes get stuck (I really achieved crashing when I successfully caught subflows exceptions and tried to continue to the next subflow) Sub flows - only get stuck in "running" state Tasks - get stuck or in "pending" or in "running" state I know, it's confusing as hell

One thing to distinguish: is the underlying process stuck or is it reflecting as "stuck" in the prefect UI?

The processes are stuck in the UI and in logs in the k8s job that runs the flow I only get is of the async problem and of the APILogWorkerThread

Hi, i'm seeing something similar. did you ever solve this?

Yeah it was a memory leak in an old prefect version, try to update it first