Thread
#prefect-community
    Naimesh Chaudhari

    Naimesh Chaudhari

    5 months ago
    How do I debug why my flow runs are sitting in submitted stage.
    Anna Geller

    Anna Geller

    5 months ago
    Naimesh Chaudhari

    Naimesh Chaudhari

    5 months ago
    OK i think its related to RBAC
    @Anna Geller so i think the helm chart automatically installs the required rbac permissions. when i set the the enable agent flag to true, which i did. When I look @ agent logs all i see is.
    [2022-04-11 19:18:28,571] INFO - agent | Completed deployment of flow run d2c7b8c0-0943-4ad0-9acc-e15ba77f8650
    INFO:agent:Completed deployment of flow run a8e9c958-ad9b-4728-a5a2-035bce653045 [2022-04-11 19:18:28,874] INFO - agent | Completed deployment of flow run a8e9c958-ad9b-4728-a5a2-035bce653045 [2022-04-11 19:18:28,970] INFO - agent | Completed deployment of flow run 2942c0ef-6e83-4de0-a395-fa8c8fd21c1b INFO:agent:Completed deployment of flow run 2942c0ef-6e83-4de0-a395-fa8c8fd21c1b [2022-04-11 19:18:28,975] INFO - agent | Completed deployment of flow run 29b83852-66cb-4f5e-b928-33309bbb4b7e INFO:agent:Completed deployment of flow run 29b83852-66cb-4f5e-b928-33309bbb4b7
    Kevin Kho

    Kevin Kho

    5 months ago
    Your logs look right. The agent needs RBAC specified but you looks seem like the permissions are there. What kind of agent are you using? Kubernetes? I think this is more related to not being to get the compute needed
    Naimesh Chaudhari

    Naimesh Chaudhari

    5 months ago
    yea it is a Kubernetes agent
    is there a way to debug if it is a resource issue?
    Kevin Kho

    Kevin Kho

    5 months ago
    There should be a pod that spun up after the job was deployed and you should be able to see logs for that or why the job didn’t start
    Naimesh Chaudhari

    Naimesh Chaudhari

    5 months ago
    thats the thing i dont see any job pods deployed
    default         traefik-7f7f89d67c-hv6lv                 1/1     Running     0          5h19m
    kube-system     aws-node-h468w                           1/1     Running     0          5h19m
    kube-system     aws-node-lnfxh                           1/1     Running     0          5h19m
    kube-system     coredns-f47955f89-625r8                  1/1     Running     0          5h23m
    kube-system     coredns-f47955f89-86kxf                  1/1     Running     0          5h23m
    kube-system     kube-proxy-hlbcw                         1/1     Running     0          5h19m
    kube-system     kube-proxy-rvtjl                         1/1     Running     0          5h19m
    mlflow          mlflow-54b98566cb-lk44j                  1/1     Running     0          5h18m
    prefectserver   prefectserver-agent-6cb7bfcd8f-57j46     1/1     Running     1          8m31s
    prefectserver   prefectserver-apollo-5d9cbdcc69-9bjww    1/1     Running     0          8m31s
    prefectserver   prefectserver-create-tenant-job-gwqrh    0/1     Completed   2          4h9m
    prefectserver   prefectserver-graphql-6fbdd7bc8d-w4ztf   1/1     Running     0          8m31s
    prefectserver   prefectserver-hasura-dbb7b87b4-tpknc     1/1     Running     0          8m31s
    prefectserver   prefectserver-towel-85659797d5-9hlzq     1/1     Running     0          8m26s
    prefectserver   prefectserver-ui-54666bcb44-fw5bc        1/1     Running     0          8m27s
    Kevin Kho

    Kevin Kho

    5 months ago
    There should be a job. Maybe run your agent with debug level logs?
    prefect agent local start --log-level=DEBUG
    Naimesh Chaudhari

    Naimesh Chaudhari

    5 months ago
    OMG! i am so stupid. I had attached the wrong service account facepalm
    Can I ask your guidance on one more thing. What is the best approach to link this to S3 Bucket? I tried it earlier and it gave me "unable to load credentials"
    Kevin Kho

    Kevin Kho

    5 months ago
    I think this is a good read that is similar, Basically you need to get the env vars in k8s
    And this for local secrets in general
    Naimesh Chaudhari

    Naimesh Chaudhari

    5 months ago
    thanks ill go through them