How do I debug why my flow runs are sitting in submitted sta

Question

How do I debug why my flow runs are sitting in submitted stage

Anna Geller · Answer

this page explains it <https discourse prefect io t why is my flow stuck in a submitted state 201>

Naimesh Chaudhari · Answer

OK i think its related to RBAC

Naimesh Chaudhari · Answer

< Anna Geller> so i think the helm chart automatically installs the required rbac permissions when i set the the enable agent flag to true which i did When I look agent logs all i see is ` 2022 04 11 19 18 28 571 INFO agent | Completed deployment of flow run d2c7b8c0 0943 4ad0 9acc e15ba77f8650` INFO agent Completed deployment of flow run a8e9c958 ad9b 4728 a5a2 035bce653045 2022 04 11 19 18 28 874 INFO agent | Completed deployment of flow run a8e9c958 ad9b 4728 a5a2 035bce653045 2022 04 11 19 18 28 970 INFO agent | Completed deployment of flow run 2942c0ef 6e83 4de0 a395 fa8c8fd21c1b INFO agent Completed deployment of flow run 2942c0ef 6e83 4de0 a395 fa8c8fd21c1b 2022 04 11 19 18 28 975 INFO agent | Completed deployment of flow run 29b83852 66cb 4f5e b928 33309bbb4b7e INFO agent Completed deployment of flow run 29b83852 66cb 4f5e b928 33309bbb4b7

Kevin Kho · Answer

Your logs look right The <https docs prefect io orchestration agents kubernetes html running in cluster|agent> needs RBAC specified but you looks seem like the permissions are there What kind of agent are you using Kubernetes I think this is more related to not being to get the compute needed

Naimesh Chaudhari · Answer

yea it is a Kubernetes agent

Naimesh Chaudhari · Answer

is there a way to debug if it is a resource issue

Kevin Kho · Answer

There should be a pod that spun up after the job was deployed and you should be able to see logs for that or why the job didn t start

Naimesh Chaudhari · Answer

thats the thing i dont see any job pods deployed

Naimesh Chaudhari · Answer

`default traefik 7f7f89d67c hv6lv 1 1 Running 0 5h19m` `kube system aws node h468w 1 1 Running 0 5h19m` `kube system aws node lnfxh 1 1 Running 0 5h19m` `kube system coredns f47955f89 625r8 1 1 Running 0 5h23m` `kube system coredns f47955f89 86kxf 1 1 Running 0 5h23m` `kube system kube proxy hlbcw 1 1 Running 0 5h19m` `kube system kube proxy rvtjl 1 1 Running 0 5h19m` `mlflow mlflow 54b98566cb lk44j 1 1 Running 0 5h18m` `prefectserver prefectserver agent 6cb7bfcd8f 57j46 1 1 Running 1 8m31s` `prefectserver prefectserver apollo 5d9cbdcc69 9bjww 1 1 Running 0 8m31s` `prefectserver prefectserver create tenant job gwqrh 0 1 Completed 2 4h9m` `prefectserver prefectserver graphql 6fbdd7bc8d w4ztf 1 1 Running 0 8m31s` `prefectserver prefectserver hasura dbb7b87b4 tpknc 1 1 Running 0 8m31s` `prefectserver prefectserver towel 85659797d5 9hlzq 1 1 Running 0 8m26s` `prefectserver prefectserver ui 54666bcb44 fw5bc 1 1 Running 0 8m27s`

Kevin Kho · Answer

There should be a job Maybe run your agent with debug level logs ```prefect agent local start log level=DEBUG```

Naimesh Chaudhari · Answer

OMG i am so stupid I had attached the wrong service account facepalm

Naimesh Chaudhari · Answer

Can I ask your guidance on one more thing What is the best approach to link this to S3 Bucket I tried it earlier and it gave me unable to load credentials

Kevin Kho · Answer

I think <https discourse prefect io t what is the correct and secure way to pass gitlab access token to a kubernetesagent using server deployed with helm chart 147|this> is a good read that is similar Basically you need to get the env vars in k8s

Kevin Kho · Answer

And <https discourse prefect io t how to set secrets e g github access token on server 70|this> for local secrets in general

Naimesh Chaudhari · Answer

thanks ill go through them

Anna Geller · Answer

this may also help <https discourse prefect io t how to assign iam permissions to a kubernetes cluster so that my flows can access other aws services such as s3 or dynamodb 341|https discourse prefect io t how to assign iam permissions to a kubernetes cluster so s can access other aws services such as s3 or dynamodb 341>