Hey all - I’ve been running prefect in production ...
# prefect-server
Hey all - I’ve been running prefect in production with no issues on
. This morning, no prefect job pods are spinning up - has anyone encountered this?
I can see the agent is running and looks like querying for flows, so I am not sure why it wouldn’t be able to create a prefect-job pod
That’s tricky. Is there a service account with the agent and could the permissions have changed? I think the agent pod should have logs on trying to spin up jobs right? If it is a resource issue, the agent should just try and you’ll get the
I don’t think any issue with a service account, as I don’t think we set it up with the helm chart. I do see a ton of logs being written to hasura, but I’m not sure how that would point to anything
Are the Flows in Scheduled or Submitted?
I feel then that Hasura wouldnt be the problem cuz the agents are able to pull from the API. I guess the test to do would be to see if you can get a pod up outside of Prefect?
yeah log is just saying deploying/deployed flow run…it’s just a massive number of hasura requests that are happening…i’ll see if I can get another pod going
so there is an interesting change in our logs…this is showing up in the slack webhook
*Link:* <http://localhost:4200/default/flow-run/flow_uuid>
instead of
*Link:* <http://prod-prefect-server-hasura.data-prod:3000/v1alpha1/graphql/default/flow_uuid>
so it seems like it might be trying to access a different endpoint? I don’t know
Kevin - thanks for the help as always - I think it is an error with another app in our kube deployment. Thanks for helping me with my pre-coffee wakeup panic haha
Oh ok sounds good