Is anyone else seeing jobs fail to start this morn...
# prefect-community
m
Is anyone else seeing jobs fail to start this morning using the hybrid hosting model? Our jobs are not getting picked up this morning but we have not seen any errors in the logs on our Agents running in our k8s cluster and did not make any changes for our k8s environment or our Agents
1
b
Hi Matt, we received your inquiry in the service console as well. Would you mind giving us an example of a flow ID that is not being picked up?
A flow run ID would be sufficient as well.
m
Hi Bianca. something seems to have cleared it up now I think but we were in a bad state for at least 12 hours. Here was one we saw not get picked up this morning 0d1e49b0-f2b8-42e9-88ef-c318bc7328b8
gratitude thank you 1
b
Hi Matt, it appears that the flow run was cancelled at
2022-06-29T14:47:21.405634+00:00
, which interrupted/cancelled all running tasks
2022-06-29T14:47:21.406375+00:00
, and then resulted in a loss of heartbeat which prompted the zombiekiller service to fail the remaining tasks runs
2022-06-29T14:48:24.749226+00:00
.
Outside of that, I don't see anything else unusual. To my knowledge, we haven't experienced any recent outages on our end either. We'll take note of this to see if any other factors could have caused this. Please do reach out if it happens again.
🙏 1
m
Thank you for the triage and follow up. I will let you know if the issue reoccurs.
👍 1
b
Here's a reference that may be helpful to rule out a service outage: https://prefect.status.io/
👍 1