Hey all I ve been running prefect in production with no issu

Question

Hey all I ve been running prefect in production with no issues on `kubernetes` This morning no prefect job pods are spinning up has anyone encountered this

Sam Werbalowsky · Answer

I can see the agent is running and looks like querying for flows so I am not sure why it wouldn t be able to create a prefect job pod

Kevin Kho · Answer

That s tricky Is there a service account with the agent and could the permissions have changed I think the agent pod should have logs on trying to spin up jobs right If it is a resource issue the agent should just try and you ll get the `NotReady` pods

Sam Werbalowsky · Answer

I don t think any issue with a service account as I don t think we set it up with the helm chart I do see a ton of logs being written to hasura but I m not sure how that would point to anything

Kevin Kho · Answer

Are the Flows in Scheduled or Submitted

Sam Werbalowsky · Answer

Submitted

Kevin Kho · Answer

I feel then that Hasura wouldnt be the problem cuz the agents are able to pull from the API I guess the test to do would be to see if you can get a pod up outside of Prefect

Sam Werbalowsky · Answer

yeah log is just saying deploying deployed flow run it s just a massive number of hasura requests that are happening i ll see if I can get another pod going

Sam Werbalowsky · Answer

so there is an interesting change in our logs this is showing up in the slack webhook ` Link ` instead of ` Link `

Sam Werbalowsky · Answer

so it seems like it might be trying to access a different endpoint I don t know

Sam Werbalowsky · Answer

Kevin thanks for the help as always I think it is an error with another app in our kube deployment Thanks for helping me with my pre coffee wakeup panic haha

Kevin Kho · Answer

Oh ok sounds good