s

    Sam Werbalowsky

    7 months ago
    Hey all - I’ve been running prefect in production with no issues on
    kubernetes
    . This morning, no prefect job pods are spinning up - has anyone encountered this?
    I can see the agent is running and looks like querying for flows, so I am not sure why it wouldn’t be able to create a prefect-job pod
    Kevin Kho

    Kevin Kho

    7 months ago
    That’s tricky. Is there a service account with the agent and could the permissions have changed? I think the agent pod should have logs on trying to spin up jobs right? If it is a resource issue, the agent should just try and you’ll get the
    NotReady
    pods
    s

    Sam Werbalowsky

    7 months ago
    I don’t think any issue with a service account, as I don’t think we set it up with the helm chart. I do see a ton of logs being written to hasura, but I’m not sure how that would point to anything
    Kevin Kho

    Kevin Kho

    7 months ago
    Are the Flows in Scheduled or Submitted?
    s

    Sam Werbalowsky

    7 months ago
    Submitted
    Kevin Kho

    Kevin Kho

    7 months ago
    I feel then that Hasura wouldnt be the problem cuz the agents are able to pull from the API. I guess the test to do would be to see if you can get a pod up outside of Prefect?
    s

    Sam Werbalowsky

    7 months ago
    yeah log is just saying deploying/deployed flow run…it’s just a massive number of hasura requests that are happening…i’ll see if I can get another pod going
    so there is an interesting change in our logs…this is showing up in the slack webhook
    *Link:* <http://localhost:4200/default/flow-run/flow_uuid>
    instead of
    *Link:* <http://prod-prefect-server-hasura.data-prod:3000/v1alpha1/graphql/default/flow_uuid>
    so it seems like it might be trying to access a different endpoint? I don’t know
    Kevin - thanks for the help as always - I think it is an error with another app in our kube deployment. Thanks for helping me with my pre-coffee wakeup panic haha
    Kevin Kho

    Kevin Kho

    7 months ago
    Oh ok sounds good