Hey all - I’ve been running prefect in production with no issues on `kubernetes`. This morning, no p...

Sam Werbalowsky

02/10/2022, 2:24 PM

Hey all - I’ve been running prefect in production with no issues on

kubernetes

. This morning, no prefect job pods are spinning up - has anyone encountered this?

Sam Werbalowsky

02/10/2022, 2:31 PM

I can see the agent is running and looks like querying for flows, so I am not sure why it wouldn’t be able to create a prefect-job pod

Kevin Kho

02/10/2022, 2:33 PM

That’s tricky. Is there a service account with the agent and could the permissions have changed? I think the agent pod should have logs on trying to spin up jobs right? If it is a resource issue, the agent should just try and you’ll get the

NotReady

pods

Sam Werbalowsky

02/10/2022, 2:37 PM

I don’t think any issue with a service account, as I don’t think we set it up with the helm chart. I do see a ton of logs being written to hasura, but I’m not sure how that would point to anything

Kevin Kho

02/10/2022, 2:39 PM

Are the Flows in Scheduled or Submitted?

Sam Werbalowsky

02/10/2022, 2:39 PM

Submitted

Kevin Kho

02/10/2022, 2:41 PM

I feel then that Hasura wouldnt be the problem cuz the agents are able to pull from the API. I guess the test to do would be to see if you can get a pod up outside of Prefect?

Sam Werbalowsky

02/10/2022, 2:43 PM

yeah log is just saying deploying/deployed flow run…it’s just a massive number of hasura requests that are happening…i’ll see if I can get another pod going

Sam Werbalowsky

02/10/2022, 2:49 PM

so there is an interesting change in our logs…this is showing up in the slack webhook

*Link:* <http://localhost:4200/default/flow-run/flow_uuid>

instead of

*Link:* <http://prod-prefect-server-hasura.data-prod:3000/v1alpha1/graphql/default/flow_uuid>

Sam Werbalowsky

02/10/2022, 2:49 PM

so it seems like it might be trying to access a different endpoint? I don’t know

Sam Werbalowsky

02/10/2022, 2:57 PM

Kevin - thanks for the help as always - I think it is an error with another app in our kube deployment. Thanks for helping me with my pre-coffee wakeup panic haha

Kevin Kho

02/10/2022, 2:57 PM

Oh ok sounds good

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.