Hey I am receiving this error after running Also the agent s

Question

Hey I am receiving this error after running Also the agent serverless kubernetes cluster is not showing up in the Prefect UI dashboard < Scott Zelenka> and < Pedro Martins> have faced similar issues before linking their previous conversations here and `$ prefect agent kubernetes install t lt MY TOKEN gt rbac | kubectl apply f ` Warning Unhealthy kubelet Liveness probe failed 8080 api health Logs of the failing pod made to restart by K8s `Warning LoggingDisabled 48m fargate scheduler Disabled logging because aws logging configmap was not found configmap aws logging not found` `Normal Scheduled 48m fargate scheduler Successfully assigned default prefect agent 7bcdb4f975 pftnw to fargate ip 10 0 231 193 ap southeast 1 compute internal` `Normal Created 43m x4 over 47m kubelet Created container agent` `Normal Started 43m x4 over 47m kubelet Started container agent` `Normal Killing 42m x2 over 45m kubelet Container agent failed liveness probe will be restarted` `Normal Pulling 41m x5 over 48m kubelet Pulling image prefecthq prefect 0 14 0 python3 6 ` `Normal Pulled 41m x5 over 47m kubelet Successfully pulled image prefecthq prefect 0 14 0 python3 6 ` `Warning Unhealthy 12m x13 over 46m kubelet Liveness probe failed Get dial tcp 10 0 231 193 8080 connect connection refused` `Warning BackOff 3m3s x113 over 43m kubelet Back off restarting failed container`

Sagun Garg · Answer

Here is what I get when I describe my pod Warning Unhealthy 4m43s x27 over 94m kubelet Liveness probe failed Get dial tcp 10 0 231 193 8080 connect connection refused Warning BackOff 18s x270 over 90m kubelet Back off restarting failed container `Name prefect agent 7bcdb4f975 pftnw` `Namespace default` `Priority 2000001000` `Priority Class Name system node critical` `Node fargate ip 10 0 231 193 ap southeast 1 compute internal 10 0 231 193` `Start Time Mon 21 Dec 2020 14 37 38 +0800` `Labels app=prefect agent` ` ` `pod template hash=7bcdb4f975` `Annotations CapacityProvisioned 0 25vCPU 0 5GB` `Logging LoggingDisabled LOGGING CONFIGMAP NOT FOUND` ` eks privileged` `Status Running` `IP 10 0 231 193` `IPs ` `IP 10 0 231 193` `Controlled By ReplicaSet prefect agent 7bcdb4f975` `Containers ` `agent ` `Container ID 1` `Image prefecthq prefect 0 14 0 python3 6` `Image ID ` `Port lt none gt ` `Host Port lt none gt ` `Command ` ` bin bash` ` c` `Args ` `prefect agent kubernetes start` `State Waiting` `Reason CrashLoopBackOff` `Last State Terminated` `Reason Error` `Exit Code 1` `Started Mon 21 Dec 2020 16 07 26 +0800` `Finished Mon 21 Dec 2020 16 08 48 +0800` `Ready False` `Restart Count 26` `Limits ` `cpu 100m` `memory 128Mi` `Requests ` `cpu 100m` `memory 128Mi` `Liveness http get http 8080 api health delay=40s timeout=1s period=40s success=1 failure=2` `Environment ` `PREFECT CLOUD AGENT AUTH TOKEN gNZkGzQJohYgunuMh okKw` `PREFECT CLOUD API ` `NAMESPACE default` `IMAGE PULL SECRETS ` `PREFECT CLOUD AGENT LABELS ` `JOB MEM REQUEST ` `JOB MEM LIMIT ` `JOB CPU REQUEST ` `JOB CPU LIMIT ` `IMAGE PULL POLICY ` `SERVICE ACCOUNT NAME ` `PREFECT BACKEND cloud` `PREFECT CLOUD AGENT AGENT ADDRESS http 8080` `Mounts ` ` var run secrets kubernetes io serviceaccount from default token 4tqz5 ro ` `Conditions ` `Type Status` `Initialized True` `Ready False` `ContainersReady False` `PodScheduled True` `Volumes ` `default token 4tqz5 ` `Type Secret a volume populated by a Secret ` `SecretName default token 4tqz5` `Optional false` `QoS Class Guaranteed` `Node Selectors lt none gt ` `Tolerations op=Exists for 300s` ` op=Exists for 300s` `Events ` `Type Reason Age From Message` ` ` `Warning Unhealthy 4m43s x27 over 94m kubelet Liveness probe failed Get dial tcp 10 0 231 193 8080 connect connection refused` `Warning BackOff 18s x270 over 90m kubelet Back off restarting failed container`

Sagun Garg · Answer

Error Logs using the pod logs $ kubectl n lt namespace name gt logs p lt pod name gt `Traceback most recent call last ` `File usr local lib python3 6 site packages urllib3 connection py line 170 in new conn` ` self dns host self port self timeout extra kw` `File usr local lib python3 6 site packages urllib3 util connection py line 73 in create connection` `for res in socket getaddrinfo host port family socket SOCK STREAM ` `File usr local lib python3 6 socket py line 745 in getaddrinfo` `for res in socket getaddrinfo host port family type proto flags ` `socket gaierror Errno 3 Temporary failure in name resolution` `During handling of the above exception another exception occurred ` `Traceback most recent call last ` `File usr local lib python3 6 site packages urllib3 connectionpool py line 706 in urlopen` `chunked=chunked ` `File usr local lib python3 6 site packages urllib3 connectionpool py line 382 in make request` `self validate conn conn ` `File usr local lib python3 6 site packages urllib3 connectionpool py line 1010 in validate conn` `conn connect ` `File usr local lib python3 6 site packages urllib3 connection py line 353 in connect` `conn = self new conn ` `File usr local lib python3 6 site packages urllib3 connection py line 182 in new conn` `self Failed to establish a new connection s e` `urllib3 exceptions NewConnectionError lt urllib3 connection HTTPSConnection object at 0x7fbb61ef5fd0 gt Failed to establish a new connection Errno 3 Temporary failure in name resolution` `During handling of the above exception another exception occurred ` `Traceback most recent call last ` `File usr local lib python3 6 site packages requests adapters py line 449 in send` `timeout=timeout` `File usr local lib python3 6 site packages urllib3 connectionpool py line 796 in urlopen` ` response kw` `File usr local lib python3 6 site packages urllib3 connectionpool py line 796 in urlopen` ` response kw` `File usr local lib python3 6 site packages urllib3 connectionpool py line 796 in urlopen` ` response kw` ` Previous line repeated 3 more times ` `File usr local lib python3 6 site packages urllib3 connectionpool py line 756 in urlopen` `method url error=e pool=self stacktrace=sys exc info 2 ` `File usr local lib python3 6 site packages urllib3 util retry py line 573 in increment` `raise MaxRetryError pool url error or ResponseError cause ` `urllib3 exceptions MaxRetryError HTTPSConnectionPool host= port=443 Max retries exceeded with url Caused by NewConnectionError lt urllib3 connection HTTPSConnection object at 0x7fbb61ef5fd0 gt Failed to establish a new connection Errno 3 Temporary failure in name resolution ` `During handling of the above exception another exception occurred ` `Traceback most recent call last ` `File usr local bin prefect line 8 in lt module gt ` `sys exit cli ` `File usr local lib python3 6 site packages click core py line 829 in call ` `return self main args kwargs ` `File usr local lib python3 6 site packages click core py line 782 in main` `rv = self invoke ctx ` `File usr local lib python3 6 site packages click core py line 1259 in invoke` `return process result sub ctx command invoke sub ctx ` `File usr local lib python3 6 site packages click core py line 1259 in invoke` `return process result sub ctx command invoke sub ctx ` `File usr local lib python3 6 site packages click core py line 1259 in invoke` `return process result sub ctx command invoke sub ctx ` `File usr local lib python3 6 site packages click core py line 1066 in invoke` `return ctx invoke self callback ctx params ` `File usr local lib python3 6 site packages click core py line 610 in invoke` `return callback args kwargs ` `File usr local lib python3 6 site packages prefect cli agent py line 282 in start` `start agent KubernetesAgent image pull secrets=image pull secrets kwargs ` `File usr local lib python3 6 site packages prefect cli agent py line 109 in start agent` `agent start ` `File usr local lib python3 6 site packages prefect agent agent py line 221 in start` `self verify token self client get auth token ` `File usr local lib python3 6 site packages prefect agent agent py line 175 in verify token` `result = self client graphql query= query auth info api token scope ` `File usr local lib python3 6 site packages prefect client client py line 303 in graphql` `retry on api error=retry on api error ` `File usr local lib python3 6 site packages prefect client client py line 219 in post` `retry on api error=retry on api error ` `File usr local lib python3 6 site packages prefect client client py line 445 in request` `session=session method=method url=url params=params headers=headers` `File usr local lib python3 6 site packages prefect client client py line 345 in send request` `response = url headers=headers json=params timeout=30 ` `File usr local lib python3 6 site packages requests sessions py line 590 in post` `return self request POST url data=data json=json kwargs ` `File usr local lib python3 6 site packages requests sessions py line 542 in request` `resp = self send prep send kwargs ` `File usr local lib python3 6 site packages requests sessions py line 655 in send` `r = adapter send request kwargs ` `File usr local lib python3 6 site packages requests adapters py line 516 in send` `raise ConnectionError e request=request ` `requests exceptions ConnectionError HTTPSConnectionPool host= port=443 Max retries exceeded with url Caused by NewConnectionError lt urllib3 connection HTTPSConnection object at 0x7fbb61ef5fd0 gt Failed to establish a new connection Errno 3 Temporary failure in name resolution `

Kyle Moon-Wright · Answer

Hey < Sagun Garg> It looks like the agent can t access the prefect API most likely due to an outbound networking issue or a config issue as described in one of the linked threads If the solution is similar to those threads you can in a variety of ways CLI Python Client Prefect Config Env Var to communicate to the Cloud API

Pedro Martins · Answer

If you are using prefect cloud the agent PREFECT CLOUD API should be pointing to it Make sure your backend is the cloud too `prefect backend cloud`

Pedro Martins · Answer

``` name PREFECT CLOUD API value <https api prefect io>```

Sagun Garg · Answer

< Kyle Moon Wright> < Pedro Martins> Thanks for your prompt reply I tried all the above including double checking on security group setting in AWS Also attaching my yaml for your reference here ```apiVersion apps v1 kind Deployment metadata labels app prefect agent name prefect agent spec replicas 1 selector matchLabels app prefect agent template metadata labels app prefect agent spec containers args prefect agent kubernetes start command bin bash c env name PREFECT CLOUD AGENT AUTH TOKEN value kXe ^$^$^$ ^BdgULWw name PREFECT CLOUD API value name NAMESPACE value default name IMAGE PULL SECRETS value name PREFECT CLOUD AGENT LABELS value name JOB MEM REQUEST value name JOB MEM LIMIT value name JOB CPU REQUEST value name JOB CPU LIMIT value name IMAGE PULL POLICY value name SERVICE ACCOUNT NAME value name PREFECT BACKEND value cloud name PREFECT CLOUD AGENT AGENT ADDRESS value image prefecthq prefect 0 14 0 python3 6 imagePullPolicy Always livenessProbe failureThreshold 2 httpGet path api health port 8080 initialDelaySeconds 40 periodSeconds 40 name agent resources limits cpu 100m memory 128Mi apiVersion kind Role metadata name prefect agent rbac namespace default rules apiGroups batch extensions resources jobs verbs apiGroups resources events pods verbs apiVersion kind RoleBinding metadata name prefect agent rbac namespace default roleRef apiGroup kind Role name prefect agent rbac subjects kind ServiceAccount name default```

Sagun Garg · Answer

< Kyle Moon Wright> < Pedro Martins> 1 I followed your steps for deployment yaml I substituted as per your guidelines 2 I double checked the security groups Caveat I am using private vpc but security group s have Inbound Outbound all traffic option enabled 3 I checked prefect cloud again I actually missed seeing the agent on cloud last time it but it shows there screenshot attached 4 I checked the version of prefect in light of new release prefect version prompts 0 14 0 on CLI since we are doing cluster+node setup on K8 has anything changed in the new version As I am yet to run the flows using RUN CONFIG and EXECUTOR pattern introduced in latest release on Friday

Pedro Martins · Answer

< Sagun Garg> liveness probeness failing happen when the kubelet cannot proper access the health check endpoint In my experience the PREFECT CLOUD AGENT AGENT ADDRESS should be `http 8080` Try modifying and let me know the results Also post the description of the pod that is crashing

Sagun Garg · Answer

< Pedro Martins> still liveliness failing inspite of the `http 8080` kubectl describe pod prefect agent f4c6f6545 jsl8t `Name prefect agent f4c6f6545 jsl8t` `Namespace default` `Priority 2000001000` `Priority Class Name system node critical` `Node fargate ip 10 0 241 95 ap southeast 1 compute internal 10 0 241 95` `Start Time Tue 22 Dec 2020 22 15 22 +0800` `Labels app=prefect agent` ` ` `pod template hash=f4c6f6545` `Annotations CapacityProvisioned 0 25vCPU 0 5GB` `Logging LoggingDisabled LOGGING CONFIGMAP NOT FOUND` ` eks privileged` `Status Running` `IP 10 0 241 95` `IPs ` `IP 10 0 241 95` `Controlled By ReplicaSet prefect agent f4c6f6545` `Containers ` `agent ` `Container ID 7` `Image prefecthq prefect 0 14 0 python3 6` `Image ID ` `Port lt none gt ` `Host Port lt none gt ` `Command ` ` bin bash` ` c` `Args ` `prefect agent kubernetes start` `State Running` `Started Tue 22 Dec 2020 22 15 41 +0800` `Ready True` `Restart Count 0` `Limits ` `cpu 100m` `memory 128Mi` `Requests ` `cpu 100m` `memory 128Mi` `Liveness http get http 8080 api health delay=40s timeout=1s period=40s success=1 failure=2` `Environment ` `PREFECT CLOUD AGENT AUTH TOKEN kXe E5u6IS2d7AUBdgULWw` `PREFECT CLOUD API ` `NAMESPACE default` `IMAGE PULL SECRETS ` `PREFECT CLOUD AGENT LABELS ` `JOB MEM REQUEST ` `JOB MEM LIMIT ` `JOB CPU REQUEST ` `JOB CPU LIMIT ` `IMAGE PULL POLICY ` `SERVICE ACCOUNT NAME ` `PREFECT BACKEND cloud` `PREFECT CLOUD AGENT AGENT ADDRESS http 8080` `Mounts ` ` var run secrets kubernetes io serviceaccount from default token 4tqz5 ro ` `Conditions ` `Type Status` `Initialized True` `Ready True` `ContainersReady True` `PodScheduled True` `Volumes ` `default token 4tqz5 ` `Type Secret a volume populated by a Secret ` `SecretName default token 4tqz5` `Optional false` `QoS Class Guaranteed` `Node Selectors lt none gt ` `Tolerations op=Exists for 300s` ` op=Exists for 300s` `Events ` `Type Reason Age From Message` ` ` `Warning LoggingDisabled 101s fargate scheduler Disabled logging because aws logging configmap was not found configmap aws logging not found` `Normal Scheduled 68s fargate scheduler Successfully assigned default prefect agent f4c6f6545 jsl8t to fargate ip 10 0 241 95 ap southeast 1 compute internal` `Normal Pulling 68s kubelet Pulling image prefecthq prefect 0 14 0 python3 6 ` `Normal Pulled 50s kubelet Successfully pulled image prefecthq prefect 0 14 0 python3 6 ` `Normal Created 49s kubelet Created container agent` `Normal Started 49s kubelet Started container agent` ` venv Saguns MacBook Pro ingress sagungargs$ kubectl get pods` `NAME READY STATUS RESTARTS AGE` `prefect agent f4c6f6545 jsl8t 1 1 Running 0 119s` ` venv Saguns MacBook Pro ingress sagungargs$ kubectl get pods` `NAME READY STATUS RESTARTS AGE` `prefect agent f4c6f6545 jsl8t 1 1 Running 0 2m9s` ` venv Saguns MacBook Pro ingress sagungargs$ kubectl describe pod prefect agent f4c6f6545 jsl8t` `Name prefect agent f4c6f6545 jsl8t` `Namespace default` `Priority 2000001000` `Priority Class Name system node critical` `Node fargate ip 10 0 241 95 ap southeast 1 compute internal 10 0 241 95` `Start Time Tue 22 Dec 2020 22 15 22 +0800` `Labels app=prefect agent` ` ` `pod template hash=f4c6f6545` `Annotations CapacityProvisioned 0 25vCPU 0 5GB` `Logging LoggingDisabled LOGGING CONFIGMAP NOT FOUND` ` eks privileged` `Status Running` `IP 10 0 241 95` `IPs ` `IP 10 0 241 95` `Controlled By ReplicaSet prefect agent f4c6f6545` `Containers ` `agent ` `Container ID 7` `Image prefecthq prefect 0 14 0 python3 6` `Image ID ` `Port lt none gt ` `Host Port lt none gt ` `Command ` ` bin bash` ` c` `Args ` `prefect agent kubernetes start` `State Running` `Started Tue 22 Dec 2020 22 15 41 +0800` `Ready True` `Restart Count 0` `Limits ` `cpu 100m` `memory 128Mi` `Requests ` `cpu 100m` `memory 128Mi` `Liveness http get http 8080 api health delay=40s timeout=1s period=40s success=1 failure=2` `Environment ` `PREFECT CLOUD AGENT AUTH TOKEN kXe E5u6IS2d7AUBdgULWw` `PREFECT CLOUD API ` `NAMESPACE default` `IMAGE PULL SECRETS ` `PREFECT CLOUD AGENT LABELS ` `JOB MEM REQUEST ` `JOB MEM LIMIT ` `JOB CPU REQUEST ` `JOB CPU LIMIT ` `IMAGE PULL POLICY ` `SERVICE ACCOUNT NAME ` `PREFECT BACKEND cloud` `PREFECT CLOUD AGENT AGENT ADDRESS http 8080` `Mounts ` ` var run secrets kubernetes io serviceaccount from default token 4tqz5 ro ` `Conditions ` `Type Status` `Initialized True` `Ready True` `ContainersReady True` `PodScheduled True` `Volumes ` `default token 4tqz5 ` `Type Secret a volume populated by a Secret ` `SecretName default token 4tqz5` `Optional false` `QoS Class Guaranteed` `Node Selectors lt none gt ` `Tolerations op=Exists for 300s` ` op=Exists for 300s` `Events ` `Type Reason Age From Message` ` ` `Warning LoggingDisabled 2m14s fargate scheduler Disabled logging because aws logging configmap was not found configmap aws logging not found` `Normal Scheduled 101s fargate scheduler Successfully assigned default prefect agent f4c6f6545 jsl8t to fargate ip 10 0 241 95 ap southeast 1 compute internal` `Normal Pulling 101s kubelet Pulling image prefecthq prefect 0 14 0 python3 6 ` `Normal Pulled 83s kubelet Successfully pulled image prefecthq prefect 0 14 0 python3 6 ` `Normal Created 82s kubelet Created container agent` `Normal Started 82s kubelet Started container agent` `Warning Unhealthy 3s kubelet Liveness probe failed Get dial tcp 10 0 241 95 8080 connect connection refused`

Sagun Garg · Answer

< Pedro Martins> Also PFA my new deployment yaml apiVersion apps v1 kind Deployment metadata labels app prefect agent name prefect agent spec replicas 1 selector matchLabels app prefect agent template metadata labels app prefect agent spec containers args prefect agent kubernetes start command bin bash c env name PREFECT CLOUD AGENT AUTH TOKEN value kXe E BdgU name PREFECT CLOUD API value name NAMESPACE value default name IMAGE PULL SECRETS value name PREFECT CLOUD AGENT LABELS value name JOB MEM REQUEST value name JOB MEM LIMIT value name JOB CPU REQUEST value name JOB CPU LIMIT value name IMAGE PULL POLICY value name SERVICE ACCOUNT NAME value name PREFECT BACKEND value cloud name PREFECT CLOUD AGENT AGENT ADDRESS value http 8080 image prefecthq prefect 0 14 0 python3 6 imagePullPolicy Always livenessProbe failureThreshold 2 httpGet path api health port 8080 initialDelaySeconds 40 periodSeconds 40 name agent resources limits cpu 100m memory 128Mi apiVersion kind Role metadata name prefect agent rbac namespace default rules apiGroups batch extensions resources jobs verbs apiGroups resources events pods verbs apiVersion kind RoleBinding metadata name prefect agent rbac namespace default roleRef apiGroup kind Role name prefect agent rbac subjects kind ServiceAccount name default