Hi, I am using google cloud run with Serverless work pool, flows takes a lot of time to run there. Is there a way to debug if it's an issue with prefect or google cloud?

Hi @Ankit, just checking to see if we're running into the same issue. My flows are 5-15 minutes

late

since a couple of days. Previously, they would run 10-90 seconds after scheduled run time, but this has increased a lot since January 12th. Initially the late run time bumped to 3 minutes. On the 13th of Jan it went up to 5, and since yesterday all the way to 15 minutes. Are you running V1 or V2 Cloud Run Push Workpool?

Yes, I can also confirm that we started seeing a lot of late runs using new push work pools.

Hey @Martijn van Elferen I am running V1 right now, will test V2 also to see if that helps. Which one are you using?

We're running V2 on GCR deployed containers.

That's right. Cloud run job takes a while to reflect on portal, that's when flow goes into

pending

stage

Google Cloud Reports no issues related to Cloud Run or Artifact Registry. What region are you running in @Ankit?

I'm in us-west1

Hmm, we're Europe-west4.

so not an issue with gcp then

Seems Prefect indeed. 🤔

hi thank you for raising. Will investigate and report back

thanks @Jake Kaplan

Same here, 10~15 minutes late since a few days (europe-west1)

thanks for raising again, we identified an issue that should be resolved. Please tag me here though if your runs continue to be later than expected

no delays now, thanks @Jake Kaplan

Hi can you give me your workspace id and the late flow run id?

workspace id - ead0d65d-79c9-471a-8542-7a89d3886313 flow run id - 855398d3-a36f-4a90-84ba-81106756957c After waiting for ~5 minutes, I cancelled the run.

I think this is something unrelated to the original post. The flow run doesn't seem to have entered late. It went to

PENDING

and I can see the cloud run job was created and execution was successfully invoked. You'd have to look at the container logs to see why python process didn't start/was delayed

Okay @Jake Kaplan I will check on my end. Thanks.