Hi Prefecters I m getting some flows stuck in submitted stat

Question

Hi Prefecters I m getting some flows stuck in submitted state perhaps one in ten or so I can t see a solution here in the Slack I m running Prefect server in Kubernetes with a Kubernetes agent more details in thread

Aiden Price · Answer

I can see a bunch of exceptions in the agent like this `urllib3 exceptions ReadTimeoutError HTTPSConnectionPool host= <http XX XXX XX XXX|XX XXX XX XXX> port=443 Read timed out read timeout=None `

Aiden Price · Answer

The IP there is actual the IP of the kubernetes service itself in the default namespace

Aiden Price · Answer

The original line in the stack trace is ```File usr local venv lib python3 8 site packages prefect agent kubernetes agent py line 433 in deploy flow self batch client create namespaced job ```

Kevin Kho · Answer

Oh man was just searching and came across this Do feel free to ping me if I don t get back to you in a day Did you get past this

Kevin Kho · Answer

Is that an apollo timeout our k8s api timeout

Aiden Price · Answer

I still see them intermittently the last one was a few hours ago How can I tell if it s Apollo or if it s k8s

Kevin Kho · Answer

Do your error look like his <https prefect community slack com archives C014Z8DPDSR p1633970004308800 thread ts=1633792814 296900 amp cid=C014Z8DPDSR|here>

Aiden Price · Answer

Hmmm not like that stack trace below ``` 2021 10 12 01 05 00 139 WARNING Prefect Kubed | Error submitting job prefect job f48950af retrying Traceback most recent call last File usr local venv lib python3 8 site packages urllib3 connectionpool py line 699 in urlopen httplib response = self make request File usr local venv lib python3 8 site packages urllib3 connectionpool py line 445 in make request WARNING Prefect Kubed Error submitting job prefect job f48950af retrying Traceback most recent call last File usr local venv lib python3 8 site packages urllib3 connectionpool py line 699 in urlopen httplib response = self make request File usr local venv lib python3 8 site packages urllib3 connectionpool py line 445 in make request six raise from e None File lt string gt line 3 in raise from File usr local venv lib python3 8 site packages urllib3 connectionpool py line 440 in make request httplib response = conn getresponse File usr local lib python3 8 http client py line 1348 in getresponse response begin File usr local lib python3 8 http client py line 316 in begin version status reason = self read status File usr local lib python3 8 http client py line 277 in read status line = str self fp readline MAXLINE + 1 iso 8859 1 File usr local lib python3 8 socket py line 669 in readinto return self sock recv into b six raise from e None File lt string gt line 3 in raise from File usr local venv lib python3 8 site packages urllib3 connectionpool py line 440 in make request httplib response = conn getresponse File usr local lib python3 8 http client py line 1348 in getresponse response begin File usr local lib python3 8 http client py line 316 in begin version status reason = self read status File usr local lib python3 8 http client py line 277 in read status line = str self fp readline MAXLINE + 1 iso 8859 1 File usr local lib python3 8 ssl py line 1241 in recv into return self read nbytes buffer File usr local lib python3 8 ssl py line 1099 in read return self sslobj read len buffer ConnectionResetError Errno 104 Connection reset by peer During handling of the above exception another exception occurred Traceback most recent call last File usr local venv lib python3 8 site packages prefect agent kubernetes agent py line 433 in deploy flow self batch client create namespaced job File usr local venv lib python3 8 site packages kubernetes client api batch v1 api py line 66 in create namespaced job return self create namespaced job with http info namespace body kwargs noqa E501 File usr local venv lib python3 8 site packages kubernetes client api batch v1 api py line 161 in create namespaced job with http info return self api client call api File usr local venv lib python3 8 site packages kubernetes client api client py line 348 in call api return self call api resource path method File usr local venv lib python3 8 site packages kubernetes client api client py line 180 in call api response data = self request File usr local venv lib python3 8 site packages kubernetes client api client py line 391 in request return url File usr local venv lib python3 8 site packages kubernetes client rest py line 274 in POST return self request POST url File usr local venv lib python3 8 site packages kubernetes client rest py line 167 in request r = self pool manager request File usr local venv lib python3 8 site packages urllib3 request py line 78 in request return self request encode body File usr local venv lib python3 8 site packages urllib3 request py line 170 in request encode body return self urlopen method url extra kw File usr local venv lib python3 8 site packages urllib3 poolmanager py line 375 in urlopen response = conn urlopen method u request uri kw File usr local venv lib python3 8 site packages urllib3 connectionpool py line 755 in urlopen retries = retries increment File usr local venv lib python3 8 site packages urllib3 util retry py line 532 in increment raise six reraise type error error stacktrace File usr local venv lib python3 8 site packages urllib3 packages six py line 769 in reraise raise value with traceback tb File usr local venv lib python3 8 site packages urllib3 connectionpool py line 699 in urlopen httplib response = self make request File usr local venv lib python3 8 site packages urllib3 connectionpool py line 445 in make request six raise from e None File lt string gt line 3 in raise from File usr local venv lib python3 8 site packages urllib3 connectionpool py line 440 in make request httplib response = conn getresponse File usr local lib python3 8 http client py line 1348 in getresponse response begin File usr local lib python3 8 http client py line 316 in begin version status reason = self read status File usr local lib python3 8 http client py line 277 in read status line = str self fp readline MAXLINE + 1 iso 8859 1 File usr local lib python3 8 socket py line 669 in readinto return self sock recv into b File usr local lib python3 8 ssl py line 1241 in recv into return self read nbytes buffer File usr local lib python3 8 ssl py line 1099 in read return self sslobj read len buffer urllib3 exceptions ProtocolError Connection aborted ConnectionResetError 104 Connection reset by peer File usr local lib python3 8 socket py line 669 in readinto return self sock recv into b File usr local lib python3 8 ssl py line 1241 in recv into return self read nbytes buffer File usr local lib python3 8 ssl py line 1099 in read return self sslobj read len buffer ConnectionResetError Errno 104 Connection reset by peer During handling of the above exception another exception occurred Traceback most recent call last File usr local venv lib python3 8 site packages prefect agent kubernetes agent py line 433 in deploy flow self batch client create namespaced job File usr local venv lib python3 8 site packages kubernetes client api batch v1 api py line 66 in create namespaced job return self create namespaced job with http info namespace body kwargs noqa E501 File usr local venv lib python3 8 site packages kubernetes client api batch v1 api py line 161 in create namespaced job with http info return self api client call api File usr local venv lib python3 8 site packages kubernetes client api client py line 348 in call api return self call api resource path method File usr local venv lib python3 8 site packages kubernetes client api client py line 180 in call api response data = self request File usr local venv lib python3 8 site packages kubernetes client api client py line 391 in request return url File usr local venv lib python3 8 site packages kubernetes client rest py line 274 in POST return self request POST url File usr local venv lib python3 8 site packages kubernetes client rest py line 167 in request r = self pool manager request File usr local venv lib python3 8 site packages urllib3 request py line 78 in request return self request encode body File usr local venv lib python3 8 site packages urllib3 request py line 170 in request encode body return self urlopen method url extra kw File usr local venv lib python3 8 site packages urllib3 poolmanager py line 375 in urlopen response = conn urlopen method u request uri kw File usr local venv lib python3 8 site packages urllib3 connectionpool py line 755 in urlopen retries = retries increment File usr local venv lib python3 8 site packages urllib3 util retry py line 532 in increment raise six reraise type error error stacktrace File usr local venv lib python3 8 site packages urllib3 packages six py line 769 in reraise raise value with traceback tb File usr local venv lib python3 8 site packages urllib3 connectionpool py line 699 in urlopen httplib response = self make request File usr local venv lib python3 8 site packages urllib3 connectionpool py line 445 in make request six raise from e None File lt string gt line 3 in raise from File usr local venv lib python3 8 site packages urllib3 connectionpool py line 440 in make request httplib response = conn getresponse File usr local lib python3 8 http client py line 1348 in getresponse response begin File usr local lib python3 8 http client py line 316 in begin version status reason = self read status File usr local lib python3 8 http client py line 277 in read status line = str self fp readline MAXLINE + 1 iso 8859 1 File usr local lib python3 8 socket py line 669 in readinto return self sock recv into b File usr local lib python3 8 ssl py line 1241 in recv into return self read nbytes buffer File usr local lib python3 8 ssl py line 1099 in read return self sslobj read len buffer urllib3 exceptions ProtocolError Connection aborted ConnectionResetError 104 Connection reset by peer ```

Kevin Kho · Answer

Looks like the k8s job is failing Hard to say why Do you have enough resources Does the pod start but then this happens It could be a situation like <https stackoverflow com questions 65438602 getting 104 connection reset error after 5 minutes|this> Is it the same flow that errors out

Aiden Price · Answer

I m using Prefect Server so all the comms between components are within the one namespace in Kubernetes My reading of that error you posted would look like using Prefect Cloud trying to reach back into a K8S cluster is that right

Kevin Kho · Answer

The guy I linked and asked if your logs matched his is on server as well If you re talking about this StackOverflow post I was more of asking if there was something being killed potentially due to going idle

Kevin Kho · Answer

I ll ping a heavy k8s user on the Prefect team and try to get some insight

Jonas · Answer

Hi < Aiden Price> I was stumbling across the exact same issue in the last days some tasks were stuck in submitted state for 15 minutes until they were finally picked up by Lazarus Are you running the kubernetes cluster on AKS Then it could be related to the Azure Loadbalancer dropping connections as described here

Aiden Price · Answer

Yes I am running on AKS So that applies to internal cluster comms between different services in the same namespace The AKS issue referred to in the Prefect issue says it s fixed but I m still getting the issue in my cluster

Jonas · Answer

Azure has given the advice to increase the idle timeout on the loadbalancer by default 4 minutes can be increased up to 30 mins however I couldn t see any effect the connection reset was always appearing after 4 minutes and few seconds inspected the packets with tcpdump ```import logging import socket import time from kubernetes import config from urllib3 connection import HTTPConnection HTTPSConnection socket setdefaulttimeout 10 logging basicConfig format= asctime s levelname 8s message s level=logging DEBUG socket options = socket SOL SOCKET socket SO KEEPALIVE 1 socket IPPROTO TCP socket TCP KEEPINTVL 30 socket IPPROTO TCP socket TCP KEEPCNT 6 socket IPPROTO TCP 0x10 30 try socket options append socket IPPROTO TCP socket TCP KEEPIDLE 120 except pass HTTPSConnection default socket options = HTTPSConnection default socket options + socket options HTTPConnection default socket options = HTTPConnection default socket options + socket options config load kube config from kubernetes import client v1 = client CoreV1Api while True Listing config maps v1 list namespaced config map default request timeout=15 OK Sleeping 6 minutes time sleep 360 ``` This is the script I used for testing Without the `socket options` I can see a retry on the next request due to the ReadTimeoutError With the `socket options` a keep alive connection is established so the connection won t be closed by the remote server I just created a PR to fix this

Kevin Kho · Answer

Wow thanks for the PR The core team will look at it

Aiden Price · Answer

Nice work very community focussed well done