https://prefect.io logo
Title
g

Garren Moore

10/05/2022, 2:49 PM
Running a very basic flow in Prefect Cloud with prefect agent deployed in EKS. In our logging infra, I’m getting
ERROR prefect.infrastructure.process - Process 'turquoise-woodlouse' exited with status code: -9
No more messaging beyond that. In Prefect Cloud UI, the flow runs indefinitely and doesn’t show this log message (level set to DEBUG). Anyone familiar with exit code -9? Can’t find any docs.
1
cc: @Taylor Curran
t

Taylor Curran

10/05/2022, 3:02 PM
Hey Garren, could you share what docs you are working off of?
g

Garren Moore

10/05/2022, 3:10 PM
I can’t find any docs on this
r

Ryan Peden

10/05/2022, 3:10 PM
I believe exit -9 tells us a Python subprocess was killed by the operating system, usually due to running out of memory.
:upvote: 1
The Python subprocess docs explain a little more here: https://docs.python.org/3/library/subprocess.html#subprocess.CompletedProcess.returncode So -9 is saying the process was terminated by Posix signal 9, i.e. SIGKILL
I think that explains why you're seeing that logged via the agent but not seeing anything in the cloud UI - SIGKILL terminates the flow process before it can report anything is wrong
g

Garren Moore

10/05/2022, 3:44 PM
🤦 thank you @Ryan Peden - it does explain the disconnect The pod itself, however, doesn’t crash with OOM 🤔 - will look into it
r

Ryan Peden

10/05/2022, 3:55 PM
It might be because it's happening to a subprocess - IIRC you will only get OOM for a pod if a container's init process gets terminated by the kernel's OOM killer. So in this case, perhaps the subprocess running the flow gets terminated by the Linux kernel, after which the agent (and the pod it is a part of) continue running
z

Zanie

10/05/2022, 4:43 PM
g

Garren Moore

10/05/2022, 4:45 PM
@Zanie 👌