Hello! All my flows are stuck in "late" state 😢

Hi Jo 👋 what kind of work pool are you using? is it a hybrid pool with a worker, or a push pool? If it's a hybrid pool, you may want to check your on your worker to make sure that it is healthy and polling for work

Hi Bianca, I’m using Cloud Run Push work pool

Gotcha gotcha. How is your work pool/work queue status looking in the UI? Is it

unhealthy

?

Oh, I couldn't check the status of the work pool. The flows finally started but with more than an hour of delay. During the "late" status, there was no trace of the jobs in Cloud Run. I think the issue was really on Prefect's side. @Ankit @Yaron Levi @Martijn van Elferen I know that you guys also use Cloud Run Push work pool, have you also experienced this?

I think we are ok. We are not seeing late runs from our side.

We saw some delay with a single deployment run last night, but subsequent runs ran on-time.

@Johan sh Is there a place in Prefect Cloud UI to see late runs, meaning the amount of such runs?

Okay thanks guys I'm not alone haha A delay of about ten minutes is acceptable, but more than an hour is problematic for some of my use cases... 🫠

nothing at our end, avg lateness under a minute

Hello I'll take advantage of this thread to ask again for my problem (https://prefect-community.slack.com/archives/CL09KU1K7/p1718101603428929). Last night I still have flows which accumulate in late state and which don't start. No flow could run on my production environment. Can you help me or at least point me to a solution please?

all our flows are in late stage for over 30 minutes, did someone find the issue earlier?

Same here 🙋‍♂️

Same here @Ankit. I’ve only had a single run since noon (CET), but that one crashed and a test run is already 5 min late. Nothing changed in the last 48h in terms of configuration

we run a lot of small flows, over 350 flows in Late now 😕

Everything is as usual on our side (Prefect Cloud + ECS Push). 2 min avg lateness.

I'm working on Google Cloud Run push, maybe it's the reason..?

I am on GCP and hitting the same issues

I'm also on gcp cloud run push

GCP Cloud Run push too

Hi everyone, thanks for raising! 🙇 Our engineers are investigating, and are actively working towards a fix. Here's the Cloud Status page for visibility. All updates will be posted there.