I m running prefect in kubernetes with preemptible nodes may

Question

I m running prefect in kubernetes with preemptible nodes maybe not the best idea shushing face Regardless I just noticed some undesired behavior At some point the agent fails to poll the graphql service but then it just hangs Agreed the agent should be able to access the backend and admittedly my setup might not be optimal but I don t think hanging is the right behavior when it cant access the backend What is your idea on this And can we improve this Possible suggestions Increase the retry limit to unlimited and the backoff factor Maybe more specific for the KubernetesAgent but adding health checks seems like a good idea nevertheless This allows kubernetes to monitor the agent and restart the agent when it does hang

Joël Luijmes · Answer

These were the logs ``` 2020 12 22 01 32 50 084 INFO agent | Starting KubernetesAgent with labels 2020 12 22 01 32 50 084 INFO agent | Agent documentation can be found at 2020 12 22 01 32 50 084 INFO agent | Agent connecting to the Prefect API at 2020 12 22 01 32 51 106 ERROR agent | There was an error connecting to 2020 12 22 01 32 51 106 ERROR agent | message request to failed reason connect ECONNREFUSED 10 32 6 84 4201 locations line 1 column 9 path hello extensions code INTERNAL SERVER ERROR exception message request to failed reason connect ECONNREFUSED 10 32 6 84 4201 type system errno ECONNREFUSED code ECONNREFUSED 2020 12 22 01 32 51 106 INFO agent | Waiting for flow runs 2020 12 22 01 32 52 126 ERROR agent | message request to failed reason connect ECONNREFUSED 10 32 6 84 4201 locations line 2 column 5 path get runs in queue extensions code INTERNAL SERVER ERROR exception message request to failed reason connect ECONNREFUSED 10 32 6 84 4201 type system errno ECONNREFUSED code ECONNREFUSED 2020 12 22 01 32 53 590 ERROR agent | message request to failed reason connect ECONNREFUSED 10 32 6 84 4201 locations line 2 column 5 path get runs in queue extensions code INTERNAL SERVER ERROR exception message request to failed reason connect ECONNREFUSED 10 32 6 84 4201 type system errno ECONNREFUSED code ECONNREFUSED 2020 12 22 01 32 55 655 ERROR agent | message request to failed reason connect ECONNREFUSED 10 32 6 84 4201 locations line 2 column 5 path get runs in queue extensions code INTERNAL SERVER ERROR exception message request to failed reason connect ECONNREFUSED 10 32 6 84 4201 type system errno ECONNREFUSED code ECONNREFUSED 2020 12 22 01 32 58 719 ERROR agent | message request to failed reason connect ECONNREFUSED 10 32 6 84 4201 locations line 2 column 5 path get runs in queue extensions code INTERNAL SERVER ERROR exception message request to failed reason connect ECONNREFUSED 10 32 6 84 4201 type system errno ECONNREFUSED code ECONNREFUSED 2020 12 22 01 33 03 768 ERROR agent | message request to failed reason connect ECONNREFUSED 10 32 6 84 4201 locations line 2 column 5 path get runs in queue extensions code INTERNAL SERVER ERROR exception message request to failed reason connect ECONNREFUSED 10 32 6 84 4201 type system errno ECONNREFUSED code ECONNREFUSED 2020 12 22 02 09 45 236 ERROR agent | HTTPConnectionPool host= prefect apollo prefect port=4200 Max retries exceeded with url graphql Caused by ConnectTimeoutError lt urllib3 connection HTTPConnection object at 0x7f85aeebd3d0 gt Connection to prefect apollo prefect timed out connect timeout=30 ``` For now I replicated the graphql to 2 replicas on different nodes to mitigate the problem wink But I think it would be nice if the behavior in it self was improved

Jim Crist-Harif · Answer

Hi Joel gt At some point the agent fails to poll the graphql service but then it just hangs It s not clear to me from the above logs that the agent is stuck from the above error the agent should sleep for a bit before retrying Trying locally with a faulty address this is the behavior I see it retries forever with sleeps in between as intended Are there no more logs after the above gt Maybe more specific for the KubernetesAgent but adding health checks seems like a good idea nevertheless This allows kubernetes to monitor the agent and restart the agent when it does hang We do have health checks but they d only catch a faulty process one that can t do any work at all not a hung connection The health check endpoint has to be enabled and the k8s deployment configured to use it `prefect agent kubernetes install` does this automatically See <https docs prefect io orchestration agents overview html health checks> for more info gt Increase the retry limit to unlimited and the backoff factor What should be happening is the agent does a couple of retries at the request level managed by `requests` before raising to the main agent loop This loop runs forever logging any errors and waiting in between attempts So after the last log above there should be a sleep then a retry again forever When I try locally this is the behavior I see If this isn t happening I suspect things are getting stuck at a lower level Can you provide any more information about what s triggering this the above logs indicate that the agent could never successfully connect to the backend

Joël Luijmes · Answer

Hey Jim gt Are there no more logs after the above Nope this is it The UI also reported that the agent hasn t connected in many hours thus I suspected that it was stuck after that log line gt We do have health checks but they d only catch a faulty process Ouch my bad I overlooked the liveness check in the deployment sweat smile gt What should be happening is the agent does a couple of retries at the request level managed by `requests` before raising to the main agent loop Hmm that would be the more logical behavior Unfortunately after the requests backoff errors it seems like it didn t retry it any longer I ll see if I can replicate my observed behavior if I set the graphql to 0 replicas

Jim Crist-Harif · Answer

Great If you can get some more information this would make a good issue I suspect this will take more iteration than slack can provide If you wouldn t mind opening an issue at <https github com PrefectHQ prefect issues> when you re ready that d help us work towards resolving this Thanks

Joël Luijmes · Answer

Hm not sure but can t seem to replicate it I tried following 1 replicas = 0 graphql gt yields ECONNREFUSED indefinitely well for couple minutes at least 2 replicas = 2 delete serviice gt yields ENOTFOUND indefinetly 3 Mimicked the observed logs by first setting replicas = 0 and after 10 errors delete the service Maybe it was just some weird race condition I ll keep an eye on it eyes Thanks at least slightly smiling face

Joël Luijmes · Answer

Hey < Jim Crist Harif> I thought I just ran into the same issue again agent playing dead But something else might be going on The UI reports that the agent hasn t queried in 6 hours checking the logs on the agent I see similar error connection timeout However I just manually triggered a flow and the agent does perform the flow just fine open mouth Could it be that the ping health loop breaks but that it doesn t affect the checking for work loop

Joël Luijmes · Answer

Logs ``` prefect agent 6fc45b78c7 7ztlz agent 2020 12 24 05 12 26 655 ERROR agent | message request to failed reason connect ECONNREFUSED 10 32 4 23 3000 locations line 2 column 5 path tenant extensions code INTERNAL SERVER ERROR exception message request to failed reason connect ECONNREFUSED 10 32 4 23 3000 type system errno ECONNREFUSED code ECONNREFUSED prefect agent 6fc45b78c7 7ztlz agent 2020 12 24 05 12 35 743 ERROR agent | message request to failed reason connect ECONNREFUSED 10 32 4 23 3000 locations line 2 column 5 path tenant extensions code INTERNAL SERVER ERROR exception message request to failed reason connect ECONNREFUSED 10 32 4 23 3000 type system errno ECONNREFUSED code ECONNREFUSED prefect agent 6fc45b78c7 7ztlz agent 2020 12 24 10 02 24 953 ERROR agent | HTTPConnectionPool host= prefect apollo prefect port=4200 Read timed out read timeout=30 prefect agent 6fc45b78c7 7ztlz agent 2020 12 24 10 42 23 112 INFO agent | Found 1 flow run s to submit for execution prefect agent 6fc45b78c7 7ztlz agent 2020 12 24 10 42 23 223 INFO agent | Deploying flow run 927e9eb6 e5bf 4e50 a67e 83b0f9680634```

Joël Luijmes · Answer

I ve put the agent logs on debug hopefully that gives some more info Are there also some logs at the backend about the agent I could collect

Dylan · Answer

Hey < Joël Luijmes> By any chance is your agent running a newer version of Prefect than your Prefect Server instance

Dylan · Answer

Also the team works on a rotation to help everyone in our community It s totally fine to send new replies to threads to the channel but please avoid pinging individual team members

Joël Luijmes · Answer

Uhm not sure what the best way to check the versions But it seems that the agent is running 0 14 and the core version is 0 13 19+173 GF2EEEB0A4

Joël Luijmes · Answer

I m using the latest helm chart so it should pull the latest versions Ill dive into it to see whats wrong nerd face

Joël Luijmes · Answer

Hm the default tag in the chart is latest so maybe something happend there so the images didn t align properly shrug

Dylan · Answer

The best rule of thumb is to keep your agent + server deployment ahead of your core version

Dylan · Answer

The agent and server can support old versions of core

Joël Luijmes · Answer

Okay and speaking in the helm services which do you call server or core

Dylan · Answer

Prefect Core is the flows themselves So if you re running with Kubernetes the version of Prefect that s running in the k8s jobs

Dylan · Answer

that are created by the agent

Dylan · Answer

Prefect Server includes of all of the helm services minus the agent