Rio McMahon
09/28/2022, 8:34 PMprefect.infrastructure.process - Process 'meticulous-manatee' exited with status code: -9
Any guidance?Zanie
09/28/2022, 10:26 PMRio McMahon
09/28/2022, 11:20 PMZanie
09/30/2022, 5:06 PMRio McMahon
09/30/2022, 5:33 PM01:44:36.945 | ERROR | prefect.agent - Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/96d5401d-c460-465a-873c-db373c1e0ca9/workspaces/3c95e6c2-fda6-4a21-8bf8-a35a3a5e2ba9/work_queues/f664d525-bc08-4740-a437-7c3a5d375bf8/get_runs>'
{'exception_message': 'Internal Server Error'}
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 128, in get_and_submit_flow_runs
queue_runs = await self.client.get_runs_in_work_queue(
File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 918, in get_runs_in_work_queue
response = await <http://self._client.post|self._client.post>(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
return await self.request(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)9/29/2022, 7:44:36 PM ced74f2966d243fe8482e8248523e6f3 For more information check: <https://httpstatuses.com/500>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 128, in get_and_submit_flow_runs
queue_runs = await self.client.get_runs_in_work_queue(
File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 918, in get_runs_in_work_queue
response = await <http://self._client.post|self._client.post>(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
return await self.request(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
(if that stacktrace is missing information let me know; I am copying from ECS logs which format it weird)Zanie
09/30/2022, 5:51 PMThis log is coming from the agent failing to retrieve runs to submit. This doesn’t indicate failure of any of the runs it’s launched.01:44:36.945 | ERROR | prefect.agent - Server error '500 Internal Server Error' for url '<https://api.prefect.cloud/api/accounts/96d5401d-c460-465a-873c-db373c1e0ca9/workspaces/3c95e6c2-fda6-4a21-8bf8-a35a3a5e2ba9/work_queues/f664d525-bc08-4740-a437-7c3a5d375bf8/get_runs>'
A “return code” of -9 indicates that the process was killed with SIGKILL. If you aren’t doing that yourself, the OOM killer is a likely culprit.
Rio McMahon
09/30/2022, 6:07 PMZanie
09/30/2022, 6:36 PMget_runs_in_work_queue
calltry:
queue_runs = await self.client.get_runs_in_work_queue(
id=work_queue.id, limit=10, scheduled_before=before
)
submittable_runs.extend(queue_runs)
except ObjectNotFound:
self.logger.error(
f"Work queue {work_queue.name!r} ({work_queue.id}) not found."
)
except Exception as exc:
self.logger.exception(exc)