Running a very basic flow in Prefect Cloud with pr...
# ask-communityg
Garren Moore
10/05/2022, 2:49 PMRunning a very basic flow in Prefect Cloud with prefect agent deployed in EKS. In our logging infra, I’m getting
ERROR prefect.infrastructure.process - Process 'turquoise-woodlouse' exited with status code: -9✅ 1
Garren Moore
10/05/2022, 2:53 PMcc: @Taylor Curran
t
Taylor Curran
10/05/2022, 3:02 PMHey Garren, could you share what docs you are working off of?
g
Garren Moore
10/05/2022, 3:10 PMI can’t find any docs on this
r
Ryan Peden
10/05/2022, 3:10 PMI believe exit -9 tells us a Python subprocess was killed by the operating system, usually due to running out of memory.
upvote 1
Ryan Peden
10/05/2022, 3:14 PMThe Python subprocess docs explain a little more here: https://docs.python.org/3/library/subprocess.html#subprocess.CompletedProcess.returncode
So -9 is saying the process was terminated by Posix signal 9, i.e. SIGKILL
Ryan Peden
10/05/2022, 3:16 PMI think that explains why you're seeing that logged via the agent but not seeing anything in the cloud UI - SIGKILL terminates the flow process before it can report anything is wrong
g
Garren Moore
10/05/2022, 3:44 PM🤦 thank you @Ryan Peden - it does explain the disconnect
The pod itself, however, doesn’t crash with OOM 🤔 - will look into it
r
Ryan Peden
10/05/2022, 3:55 PMIt might be because it's happening to a subprocess - IIRC you will only get OOM for a pod if a container's init process gets terminated by the kernel's OOM killer.
So in this case, perhaps the subprocess running the flow gets terminated by the Linux kernel, after which the agent (and the pod it is a part of) continue running
z
Zanie
10/05/2022, 4:43 PM
g
Garren Moore
10/05/2022, 4:45 PM@Zanie 👌