Hi Folks, my flow run is getting stuck on schedule...
# ask-communitym
Marwan Sarieddine
08/31/2020, 2:21 PMHi Folks, my flow run is getting stuck on scheduled and is not getting submitted - anyone else facing similar issues ?
I am using prefect v0.13.4, prefect cloud, with a kubernetes agent, and a static dask setup -flow is stored using S3 Storage
Here is my flow-run id: 68e72aba-f69b-4238-9478-82fa9bfa5706
Marwan Sarieddine
08/31/2020, 2:24 PMthe agent is registered and shows on the UI dashboard under the agents tab -
also here are the agent logs - it says: waiting for flow runs ...
Copy code
____ __ _ _ _
| _ \ _ __ ___ / _| ___ ___| |_ / \ __ _ ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __| / _ \ / _` |/ _ \ '_ \| __|
| __/| | | __/ _| __/ (__| |_ / ___ \ (_| | __/ | | | |_
|_| |_| \___|_| \___|\___|\__| /_/ \_\__, |\___|_| |_|\__|
|___/
[2020-08-31 14:03:51,207] INFO - ifm-k8s-agent | Starting KubernetesAgent with labels []
[2020-08-31 14:03:51,208] INFO - ifm-k8s-agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-08-31 14:03:51,208] INFO - ifm-k8s-agent | Agent connecting to the Prefect API at <https://api.prefect.io>
[2020-08-31 14:03:51,340] INFO - ifm-k8s-agent | Waiting for flow runs...Marwan Sarieddine
08/31/2020, 2:25 PMNote - I had edited the agent to update the image it was using which caused two agents of the same name to show under the agents tab - so I removed the one that had a red status - and kept the green one ...
Marwan Sarieddine
08/31/2020, 2:35 PMi then removed all the agents - registered a new one with a different name, re-registered the flow and attempted to run - but I am still facing the same issue - flow run is stuck in on scheduled
j
Jeremiah
08/31/2020, 2:51 PM
Hi Marwan, can you double check that the agents have the same labels as the flow? I’m not seeing anything immediately odd on the Cloud side about that run
Jeremiah
08/31/2020, 2:52 PM
Some good news - we are actively working on improved agent communication / diagnostics to help with this sort of situation
m
Marwan Sarieddine
08/31/2020, 2:55 PMHi @Jeremiah - thank you for the prompt response
yes the agent has the same labels as the flow - and the agent was working fine earlier (yesterday)
Marwan Sarieddine
08/31/2020, 2:55 PMah one sec - I dont see the label show up on the dashboard
Marwan Sarieddine
08/31/2020, 2:55 PMhmmm
Marwan Sarieddine
08/31/2020, 3:02 PMthank you - somehow we have modified the agent k8s spec and lost the label
👍 1
4 Views