Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
pacc-june-14-2023
geo-japan
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
m
Marwan Sarieddine
08/31/2020, 2:21 PMHi Folks, my flow run is getting stuck on scheduled and is not getting submitted - anyone else facing similar issues ?
I am using prefect v0.13.4, prefect cloud, with a kubernetes agent, and a static dask setup -flow is stored using S3 Storage
Here is my flow-run id: 68e72aba-f69b-4238-9478-82fa9bfa5706
the agent is registered and shows on the UI dashboard under the agents tab -
also here are the agent logs - it says: waiting for flow runs ...
____ __ _ _ _
| _ \ _ __ ___ / _| ___ ___| |_ / \ __ _ ___ _ __ | |_
| |_) | '__/ _ \ |_ / _ \/ __| __| / _ \ / _` |/ _ \ '_ \| __|
| __/| | | __/ _| __/ (__| |_ / ___ \ (_| | __/ | | | |_
|_| |_| \___|_| \___|\___|\__| /_/ \_\__, |\___|_| |_|\__|
|___/
[2020-08-31 14:03:51,207] INFO - ifm-k8s-agent | Starting KubernetesAgent with labels []
[2020-08-31 14:03:51,208] INFO - ifm-k8s-agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2020-08-31 14:03:51,208] INFO - ifm-k8s-agent | Agent connecting to the Prefect API at <https://api.prefect.io>
[2020-08-31 14:03:51,340] INFO - ifm-k8s-agent | Waiting for flow runs...Note - I had edited the agent to update the image it was using which caused two agents of the same name to show under the agents tab - so I removed the one that had a red status - and kept the green one ...
i then removed all the agents - registered a new one with a different name, re-registered the flow and attempted to run - but I am still facing the same issue - flow run is stuck in on scheduled
j
Jeremiah
08/31/2020, 2:51 PMHi Marwan, can you double check that the agents have the same labels as the flow? I’m not seeing anything immediately odd on the Cloud side about that run
Some good news - we are actively working on improved agent communication / diagnostics to help with this sort of situation
m
Marwan Sarieddine
08/31/2020, 2:55 PMHi @Jeremiah - thank you for the prompt response
yes the agent has the same labels as the flow - and the agent was working fine earlier (yesterday)
ah one sec - I dont see the label show up on the dashboard
hmmm
thank you - somehow we have modified the agent k8s spec and lost the label
👍 1