Hello, with several long running flows I get an error like:

prefect.infrastructure.process - Process 'meticulous-manatee' exited with status code: -9

Any guidance?

Is this on Linux? -9 is a weird status code. Do you have output from the process?

It is running in a docker container on ECS which uses the prefect 2.0 as the base image. FWIW the flow doesn’t show up as crashed/failed on the docker UI

I don’t know. I’d basically just be trying to google information about your situation and a status code of -9. That’s not a code we’ll ever set.

RE logs - that will just show up with no python related stack trace; it will be logging normally and then just randomly die with that above message. This does occur after a flow runs for a long time (>1 hour with lots of logging; probably a log message every 20s on average). Note that the prefect UI does not reflect the exit status with a “Crashed” label and the flow keeps running while block any scheduled flows. This error might be related (?) https://prefect-community.slack.com/archives/C03D12VV4NN/p1659559088670819 which is why I am mentioning logging. Additionally I am also seeing long running flows (similar length/amount of logging) that will also die in the container with this error but are not reflected on the prefect 2 UI (they continue to show as running):

01:44:36.945 | ERROR | prefect.agent - Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/96d5401d-c460-465a-873c-db373c1e0ca9/workspaces/3c95e6c2-fda6-4a21-8bf8-a35a3a5e2ba9/work_queues/f664d525-bc08-4740-a437-7c3a5d375bf8/get_runs>'
{'exception_message': 'Internal Server Error'}	
Response: {'exception_message': 'Internal Server Error'}	
For more information check: <https://httpstatuses.com/500>	
Traceback (most recent call last):	
File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 128, in get_and_submit_flow_runs	
queue_runs = await self.client.get_runs_in_work_queue(	
File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 918, in get_runs_in_work_queue	
response = await <http://self._client.post|self._client.post>(	
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post	
return await self.request(	
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request	
return await self.send(request, auth=auth, follow_redirects=follow_redirects)9/29/2022, 7:44:36 PM	ced74f2966d243fe8482e8248523e6f3 	For more information check: <https://httpstatuses.com/500>	
Traceback (most recent call last):	
File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 128, in get_and_submit_flow_runs	
queue_runs = await self.client.get_runs_in_work_queue(	
File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 918, in get_runs_in_work_queue	
response = await <http://self._client.post|self._client.post>(	
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post	
return await self.request(	
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request	
return await self.send(request, auth=auth, follow_redirects=follow_redirects)

(if that stacktrace is missing information let me know; I am copying from ECS logs which format it weird)

01:44:36.945 | ERROR | prefect.agent - Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/96d5401d-c460-465a-873c-db373c1e0ca9/workspaces/3c95e6c2-fda6-4a21-8bf8-a35a3a5e2ba9/work_queues/f664d525-bc08-4740-a437-7c3a5d375bf8/get_runs>'

This log is coming from the agent failing to retrieve runs to submit. This doesn’t indicate failure of any of the runs it’s launched.

Thanks for info on the OOM stack overflow post; I will investigate. On the prefect error: will that 1. kill the prefect agent and, 2. if so do you have any thoughts on ways to restart an agent within an ECS container that dies like that? Normally I’d run stuff like this as a systemd service or similar but docker and systemd have not played nice in the past when I’ve tried stuff like that. I’m curious if there is a way to configure a healthchecker so that if the prefect agent dies the container will be restarted.

Yeah that will kill the agent. We can update the agent to be robust to failures there.