Hey all - I’ve been running prefect in production ...
# prefect-servers
Sam Werbalowsky
02/10/2022, 2:24 PMHey all - I’ve been running prefect in production with no issues on
kubernetesSam Werbalowsky
02/10/2022, 2:31 PMI can see the agent is running and looks like querying for flows, so I am not sure why it wouldn’t be able to create a prefect-job pod
k
Kevin Kho
02/10/2022, 2:33 PM
That’s tricky. Is there a service account with the agent and could the permissions have changed? I think the agent pod should have logs on trying to spin up jobs right? If it is a resource issue, the agent should just try and you’ll get the
NotReadys
Sam Werbalowsky
02/10/2022, 2:37 PMI don’t think any issue with a service account, as I don’t think we set it up with the helm chart. I do see a ton of logs being written to hasura, but I’m not sure how that would point to anything
k
Kevin Kho
02/10/2022, 2:39 PM
Are the Flows in Scheduled or Submitted?
s
Sam Werbalowsky
02/10/2022, 2:39 PMSubmitted
k
Kevin Kho
02/10/2022, 2:41 PM
I feel then that Hasura wouldnt be the problem cuz the agents are able to pull from the API. I guess the test to do would be to see if you can get a pod up outside of Prefect?
s
Sam Werbalowsky
02/10/2022, 2:43 PMyeah log is just saying deploying/deployed flow run…it’s just a massive number of hasura requests that are happening…i’ll see if I can get another pod going
Sam Werbalowsky
02/10/2022, 2:49 PMso there is an interesting change in our logs…this is showing up in the slack webhook
*Link:* <http://localhost:4200/default/flow-run/flow_uuid>*Link:* <http://prod-prefect-server-hasura.data-prod:3000/v1alpha1/graphql/default/flow_uuid>Sam Werbalowsky
02/10/2022, 2:49 PMso it seems like it might be trying to access a different endpoint? I don’t know
Sam Werbalowsky
02/10/2022, 2:57 PMKevin - thanks for the help as always - I think it is an error with another app in our kube deployment. Thanks for helping me with my pre-coffee wakeup panic haha
k
Kevin Kho
02/10/2022, 2:57 PM
Oh ok sounds good