Issue We just received an error where our Prefect agent fail

Question

Issue We just received an error where our Prefect agent failed to submit a flow run The agent continued to submit all flow runs after this specific one Error details in thread Prefect Version 2 7 8 Prefect Agent AKS

Samuel Kohlleffel · Answer

```17 59 40 136 | ERROR | prefect agent Failed to submit flow run 0a11c778 7abf 4368 bf4b b55d73f267fa to infrastructure Traceback most recent call last File usr local lib python3 10 site packages urllib3 connectionpool py line 703 in urlopen httplib response = self make request File usr local lib python3 10 site packages urllib3 connectionpool py line 449 in make request six raise from e None File lt string gt line 3 in raise from File usr local lib python3 10 site packages urllib3 connectionpool py line 444 in make request httplib response = conn getresponse File usr local lib python3 10 http client py line 1374 in getresponse response begin File usr local lib python3 10 http client py line 318 in begin version status reason = self read status File usr local lib python3 10 http client py line 287 in read status raise RemoteDisconnected Remote end closed connection without http client RemoteDisconnected Remote end closed connection without response During handling of the above exception another exception occurred Traceback most recent call last File usr local lib python3 10 site packages prefect agent py line 424 in submit run and capture errors result = await infrastructure run task status=task status File usr local lib python3 10 site packages prefect infrastructure kubernetes py line 277 in run job = await run sync in worker thread self create job manifest File usr local lib python3 10 site packages prefect utilities asyncutils py line 91 in run sync in worker thread return await anyio to thread run sync File usr local lib python3 10 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File usr local lib python3 10 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File usr local lib python3 10 site packages anyio backends asyncio py line 867 in run result = context run func args File usr local lib python3 10 site packages prefect infrastructure kubernetes py line 622 in create job job = batch client create namespaced job self namespace job manifest File usr local lib python3 10 site packages kubernetes client api batch v1 api py line 210 in create namespaced job return self create namespaced job with http info namespace body kwargs noqa E501 File usr local lib python3 10 site packages kubernetes client api batch v1 api py line 309 in create namespaced job with http info return self api client call api File usr local lib python3 10 site packages kubernetes client api client py line 348 in call api return self call api resource path method File usr local lib python3 10 site packages kubernetes client api client py line 180 in call api response data = self request File usr local lib python3 10 site packages kubernetes client api client py line 391 in request return <http self rest client POST|self rest client POST> url File usr local lib python3 10 site packages kubernetes client rest py line 276 in POST return self request POST url File usr local lib python3 10 site packages kubernetes client rest py line 169 in request r = self pool manager request File usr local lib python3 10 site packages urllib3 request py line 78 in request return self request encode body File usr local lib python3 10 site packages urllib3 request py line 170 in request encode body return self urlopen method url extra kw File usr local lib python3 10 site packages urllib3 poolmanager py line 376 in urlopen response = conn urlopen method u request uri kw File usr local lib python3 10 site packages urllib3 connectionpool py line 787 in urlopen retries = retries increment File usr local lib python3 10 site packages urllib3 util retry py line 550 in increment raise six reraise type error error stacktrace File usr local lib python3 10 site packages urllib3 packages six py line 769 in reraise raise value with traceback tb File usr local lib python3 10 site packages urllib3 connectionpool py line 703 in urlopen httplib response = self make request File usr local lib python3 10 site packages urllib3 connectionpool py line 449 in make request six raise from e None File lt string gt line 3 in raise from File usr local lib python3 10 site packages urllib3 connectionpool py line 444 in make request httplib response = conn getresponse File usr local lib python3 10 http client py line 1374 in getresponse response begin File usr local lib python3 10 http client py line 318 in begin version status reason = self read status File usr local lib python3 10 http client py line 287 in read status raise RemoteDisconnected Remote end closed connection without urllib3 exceptions ProtocolError Connection aborted RemoteDisconnected Remote end closed connection without response 17 59 40 144 | INFO | prefect agent Completed submission of flow run 0a11c778 7abf 4368 bf4b b55d73f267fa 17 59 42 045 | WARNING | urllib3 connectionpool Retrying Retry total=2 connect=None read=None redirect=None status=None after connection broken by ProtocolError Connection aborted RemoteDisconnected Remote end closed connection without response api v1 namespaces default pods papaya cockatoo 9ckbx r7gsg log follow=True```

Samuel Kohlleffel · Answer

Shortly before this issue one of our Prefect jobs crashed as well ```17 54 01 860 | INFO | prefect agent Submitting flow run c138b823 54b0 463e baec a363870cafcf 17 54 02 365 | INFO | prefect infrastructure kubernetes job Job spry limpet 764c2 Pod has status Pending 17 54 02 428 | INFO | prefect agent Completed submission of flow run c138b823 54b0 463e baec a363870cafcf 17 55 02 362 | ERROR | prefect infrastructure kubernetes job Job spry limpet 764c2 Pod never started 17 55 02 495 | INFO | prefect agent Reported flow run c138b823 54b0 463e baec a363870cafcf as crashed Flow run infrastructure exited with non zero status code 1 ```