Thanks for flagging this — I know LATE runs with no Cloud Run jobs is stressful, especially when it affects stakeholders. Short answer: we’re not seeing a widespread Prefect-side incident right now specific to Cloud Run push pools, but it can be either side (Prefect submission vs. GCP API/permissions/quotas). Let’s quickly isolate which. Immediate checks - Prefect status: Prefect Status - GCP status (filter for your region + Cloud Run): Google Cloud Status Quick Prefect-side triage (Prefect 3.x) 1) Verify the work pool isn’t paused and has capacity

prefect work-pool inspect "<your-pool>"
prefect work-queue ls

- Confirm

is_paused=False

- Look for pool or queue concurrency limits being hit 2) Preview upcoming scheduled work (does Prefect think it should be running?)

prefect work-pool preview "<your-pool>" --hours 1

3) Sanity-check infra provisioning (helps surface auth/template issues fast)

prefect work-pool provision-infrastructure "<your-pool>"

- If this fails, it’s usually a credentials, permissions, or job template issue 4) Open one of the LATE flow runs in the UI and check: - Events tab: do you see “Submitted infrastructure” or any “Failed to submit infrastructure” messages? - If there’s a submission error, it often includes the GCP error (e.g., PermissionDenied, quota, invalid region) GCP-side triage - Check Cloud Audit Logs for Cloud Run Jobs API calls at the expected times. If you see no calls, the submission likely failed before reaching GCP. Suggested Logs Explorer filter:

protoPayload.serviceName="<http://run.googleapis.com|run.googleapis.com>"
(protoPayload.methodName="google.cloud.run.v2.Jobs.CreateJob" OR protoPayload.methodName="google.cloud.run.v2.Jobs.RunJob")
timestamp >= "last 2 hours"

- Verify quotas in your region (Cloud Run Jobs executions, container registry/pull, IAM policy checks) - Confirm the service account used by the pool still has required roles (e.g., Cloud Run Admin or granular perms to create/run jobs, and Artifact Registry read if pulling images) - If credentials were rotated/disabled in the last hour (service account key, Workload Identity Federation), that will cause silent submission failures that show up as LATE runs with no Cloud Run jobs A few targeted questions to help us pinpoint the issue fast - Are you on Prefect Cloud or a self-hosted server? - Work pool name and type (Cloud Run v2 push), region, and whether you’re using a service account key or WIF - Example flow run ID(s) stuck in LATE - Any changes in the last hour (service account/keys, base job template, image, region, quotas)? - Do you see any “Failed to submit infrastructure” messages in the flow run Events? If you can share your org/workspace and a couple of run IDs via DM, we can check from our side and escalate if needed. References - GCP worker guide (Cloud Run push pools): https://docs-3.prefect.io/integrations/prefect-gcp/gcp-worker-guide If you need a short-term workaround to get data flowing while we debug, consider: - Temporarily running the same deployment in a different pool (e.g., a Docker or Kubernetes worker you control), or - Manually triggering the container in Cloud Run with the same image/command to unblock the pipeline I’m standing by — send the details and I’ll help dig in right away.

Thanks for the extra detail. If GCP audit logs show no Cloud Run Jobs API calls and your Prefect flow run Events don’t show any “submitted infrastructure” or “failed to submit infrastructure” messages, that strongly suggests the runs aren’t making it to the submission step. That usually points to work pool/queue/deployment gating (paused or concurrency limits) or a transient issue in the submission path. Let’s quickly isolate. Targeted checks (safe to run now) - Work pool/queue state and limits

prefect work-pool inspect "<your-pool>"
prefect work-queue ls
# If you see your queue, inspect it:
prefect work-queue inspect "<queue-id>"

Confirm: - Pool is not paused (

is_paused=False

) - No pool or queue concurrency limit of 0, and current usage is below the limit - Queue is not paused - Preview scheduled work (does Prefect think these runs should be active?)

prefect work-pool preview "<your-pool>" --hours 1

- Deployment is pointing at the expected pool/queue and not paused

prefect deployment inspect "<flow-name>/<deployment-name>"

Look for: -

work_pool_name

matches your Cloud Run push pool - No deployment-level concurrency limit blocking runs - Quick infra validation (local preflight)

prefect work-pool provision-infrastructure "<your-pool>"

This won’t submit a flow run, but it helps surface base job template or credential issues immediately. If this fails locally, we likely have a config problem; if it passes and there are still no submission events on flow runs, that implicates the server-side submission path. If all the above looks normal - Please share (DM is fine): your org/workspace, pool name, and 2–3 example flow run IDs stuck in LATE, plus the timeframe/region. I can check from our side and escalate to on-call if needed. Common causes that match your symptoms - Pool or queue paused, or concurrency limits effectively at 0 (including newly set limits) - Deployment moved to a different pool or queue unintentionally - Base job template or credentials were changed/removed, but if that were the case we usually still see “failed to submit infrastructure” events — the absence of those suggests submission didn’t even start Short-term workarounds to restore data availability - Move the deployment to a worker-based pool temporarily and start a worker to clear the backlog: - Create a worker pool (Docker/Kubernetes/etc.) - Update the deployment to that pool and run a worker:

prefect worker start -p "<your-worker-pool>"

- Or create a fresh Cloud Run push pool and switch a single test deployment to it to see if submission resumes Status/links - Prefect status: Prefect Status - GCP status: Google Cloud Status - GCP guide for Prefect (permissions, setup): GCP worker guide I know this is urgent — if you can send the org/workspace + run IDs, I’ll dig in and escalate immediately.