https://prefect.io logo
m

Marwan Sarieddine

08/31/2020, 2:21 PM
Hi Folks, my flow run is getting stuck on scheduled and is not getting submitted - anyone else facing similar issues ? I am using prefect v0.13.4, prefect cloud, with a kubernetes agent, and a static dask setup -flow is stored using S3 Storage Here is my flow-run id: 68e72aba-f69b-4238-9478-82fa9bfa5706
the agent is registered and shows on the UI dashboard under the agents tab - also here are the agent logs - it says: waiting for flow runs ...
Copy code
____            __           _        _                    _
|  _ \ _ __ ___ / _| ___  ___| |_     / \   __ _  ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __|   / _ \ / _` |/ _ \ '_ \| __|
|  __/| | |  __/  _|  __/ (__| |_   / ___ \ (_| |  __/ | | | |_
|_|   |_|  \___|_|  \___|\___|\__| /_/   \_\__, |\___|_| |_|\__|
                                           |___/

[2020-08-31 14:03:51,207] INFO - ifm-k8s-agent | Starting KubernetesAgent with labels []
[2020-08-31 14:03:51,208] INFO - ifm-k8s-agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-08-31 14:03:51,208] INFO - ifm-k8s-agent | Agent connecting to the Prefect API at <https://api.prefect.io>
[2020-08-31 14:03:51,340] INFO - ifm-k8s-agent | Waiting for flow runs...
Note - I had edited the agent to update the image it was using which caused two agents of the same name to show under the agents tab - so I removed the one that had a red status - and kept the green one ...
i then removed all the agents - registered a new one with a different name, re-registered the flow and attempted to run - but I am still facing the same issue - flow run is stuck in on scheduled
j

Jeremiah

08/31/2020, 2:51 PM
Hi Marwan, can you double check that the agents have the same labels as the flow? I’m not seeing anything immediately odd on the Cloud side about that run
Some good news - we are actively working on improved agent communication / diagnostics to help with this sort of situation
m

Marwan Sarieddine

08/31/2020, 2:55 PM
Hi @Jeremiah - thank you for the prompt response yes the agent has the same labels as the flow - and the agent was working fine earlier (yesterday)
ah one sec - I dont see the label show up on the dashboard
hmmm
thank you - somehow we have modified the agent k8s spec and lost the label
👍 1