Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi,
I am using google cloud run with Serverless work pool, flows takes a lot of time to run there. Is there a way to debug if it's an issue with prefect or google cloud?

Hi <@U04GNEJ20BC>, just checking to see if we're running into the same issue. My flows are 5-15 minutes `late` since a couple of days. Previously, they would run 10-90 seconds after scheduled run time, but this has increased a lot since January 12th. Initially the late run time bumped to 3 minutes. On the 13th of Jan it went up to 5, and since yesterday all the way to 15 minutes. Are you running V1 or V2 Cloud Run Push Workpool?

Yes, I can also confirm that we started seeing a lot of late runs using new push work pools.

Hey <@U05SKCH6928>
I am running V1 right now, will test V2 also to see if that helps. Which one are you using?

We're running V2 on GCR deployed containers.

In our case it takes about 6.5 minutes in `late` stage before going to `pending` . `Pending` to `running` is also at least 6 minutes.

That's right. Cloud run job takes a while to reflect on portal, that's when flow goes into `pending` stage

<https://status.cloud.google.com/regional/europe|Google Cloud Reports no issues> related to Cloud Run or Artifact Registry. What region are you running in <@U04GNEJ20BC>?

CleanShot 2024-01-15 at 16.11.34@2x.jpg

hi thank you for raising. Will investigate and report back

Capture d’écran 2024-01-15 à 15.52.08.png

Same here, 10~15 minutes late since a few days (europe-west1)

thanks for raising again, we identified an issue that should be resolved. Please tag me here though if your runs continue to be later than expected

Hey <@U02EBC3N05N>, again experiencing this. Job is not yet created in cloud run, been 5 minutes already. Shows late in prefect cloud.

Hi can you give me your workspace id and the late flow run id?

workspace id - ead0d65d-79c9-471a-8542-7a89d3886313
flow run id - 855398d3-a36f-4a90-84ba-81106756957c

After waiting for ~5 minutes, I cancelled the run.

I think this is something unrelated to the original post. The flow run doesn't seem to have entered late. It went to `PENDING` and I can see the cloud run job was created and execution was successfully invoked. You'd have to look at the container logs to see why python process didn't start/was delayed

Okay <@U02EBC3N05N> I will check on my end. Thanks.