https://prefect.io
Join Slack
How do I debug why my flow runs are sitting in submitted stage.
# ask-community
n
How do I debug why my flow runs are sitting in submitted stage.
a
n
OK i think its related to RBAC
👍 1
@Anna Geller so i think the helm chart automatically installs the required rbac permissions. when i set the the enable agent flag to true, which i did. When I look @ agent logs all i see is. 
[2022-04-11 19:18:28,571] INFO - agent | Completed deployment of flow run d2c7b8c0-0943-4ad0-9acc-e15ba77f8650
INFOagentCompleted deployment of flow run a8e9c958-ad9b-4728-a5a2-035bce653045 [2022-04-11 191828,874] INFO - agent | Completed deployment of flow run a8e9c958-ad9b-4728-a5a2-035bce653045 [2022-04-11 191828,970] INFO - agent | Completed deployment of flow run 2942c0ef-6e83-4de0-a395-fa8c8fd21c1b INFOagentCompleted deployment of flow run 2942c0ef-6e83-4de0-a395-fa8c8fd21c1b [2022-04-11 191828,975] INFO - agent | Completed deployment of flow run 29b83852-66cb-4f5e-b928-33309bbb4b7e INFOagentCompleted deployment of flow run 29b83852-66cb-4f5e-b928-33309bbb4b7
👍 1
k
Your logs look right. The agent needs RBAC specified but you looks seem like the permissions are there. What kind of agent are you using? Kubernetes? I think this is more related to not being to get the compute needed
n
yea it is a Kubernetes agent
is there a way to debug if it is a resource issue?
k
There should be a pod that spun up after the job was deployed and you should be able to see logs for that or why the job didn’t start
n
thats the thing i dont see any job pods deployed
default         traefik-7f7f89d67c-hv6lv                 1/1     Running     0          5h19m
kube-system     aws-node-h468w                           1/1     Running     0          5h19m
kube-system     aws-node-lnfxh                           1/1     Running     0          5h19m
kube-system     coredns-f47955f89-625r8                  1/1     Running     0          5h23m
kube-system     coredns-f47955f89-86kxf                  1/1     Running     0          5h23m
kube-system     kube-proxy-hlbcw                         1/1     Running     0          5h19m
kube-system     kube-proxy-rvtjl                         1/1     Running     0          5h19m
mlflow          mlflow-54b98566cb-lk44j                  1/1     Running     0          5h18m
prefectserver   prefectserver-agent-6cb7bfcd8f-57j46     1/1     Running     1          8m31s
prefectserver   prefectserver-apollo-5d9cbdcc69-9bjww    1/1     Running     0          8m31s
prefectserver   prefectserver-create-tenant-job-gwqrh    0/1     Completed   2          4h9m
prefectserver   prefectserver-graphql-6fbdd7bc8d-w4ztf   1/1     Running     0          8m31s
prefectserver   prefectserver-hasura-dbb7b87b4-tpknc     1/1     Running     0          8m31s
prefectserver   prefectserver-towel-85659797d5-9hlzq     1/1     Running     0          8m26s
prefectserver   prefectserver-ui-54666bcb44-fw5bc        1/1     Running     0          8m27s
k
There should be a job. Maybe run your agent with debug level logs?
Copy code
prefect agent local start --log-level=DEBUG
n
OMG! i am so stupid. I had attached the wrong service account facepalm
Can I ask your guidance on one more thing. What is the best approach to link this to S3 Bucket? I tried it earlier and it gave me "unable to load credentials"
k
I think this is a good read that is similar, Basically you need to get the env vars in k8s
👍 1
And this for local secrets in general
👍 1
n
thanks ill go through them
👍 1
a
92 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by