https://prefect.io
Join Slack
Can we get a /health + /stats endpoint on an agent so we can monitor and cycle if needed
# ask-community
r
Can we get a /health + /stats endpoint on an agent so we can monitor and cycle if needed
c
HI redsquare - is there a specific case you are seeing this being necessary?
Not saying that it’s not a bad idea, but I’m curious if there is another problem occurring that you feel this is missing
the agent natively is not running any sort of webserver, and has no listening connections as it is outbound only
however, you can add a liveness probe that tests if the prefect agent is still responsive, with something like this:
Copy code
livenessProbe:
            exec:
              command:
              - prefect
              - agent
              - '--help'
            initialDelaySeconds: 5
            periodSeconds: 5
r
hey @Christopher Boyd we had network blips on the agent overnight and 2 flows left in a pending state as a result
Do pending jobs ever recover if the agent couldnt kick the job off
do they eventually get set to failed
c
You mean pending kubernetes jobs?
or Pending prefect flow runs?
if it’s a kubernetes job that was started, taht wouldnt’ be a prefect issue at that point, it was handed off to execute to k8s and I would be looking at the logs there for why it’s stuck in pending (insufficient cpu/memory potentially, not enough nodes, etc.)
r
Yeah so we had socket errors on the agent - pending then means the agent accepted the job but has not reported back
and it will stay in that state given this scenario
c
Hrmmm, I’m not sure of what the expected behavior is in this case, I will need to check with the team and report back
do you have logs for the agent?
r
sadly not - we re-installed the agent over the course of the day - I will see what happens tonight
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by