Joseph Wibowo
11/18/2025, 5:42 PMWorker 'CloudRunWorkerV2 20b3739b-deb6-46d5-9ea5-479b01fc7516' submitting flow run '6a24cec0-11d2-417a-9531-ce9c32aedc38'
09:24:07 AM
prefect.flow_runs.worker
PREFECT_API_AUTH_STRING is provided as a plaintext environment variable. For better security, consider providing it as a secret using 'prefect_api_auth_string_secret' or 'env_from_secrets' in your base job template.
09:24:07 AM
prefect.flow_runs.worker
Creating Cloud Run JobV2 berserk-baboon-793ca844a1b348d98c2013331a8710a4
09:24:07 AM
prefect.flow_runs.worker
Current Job Condition: waiting for condition update
09:24:08 AM
prefect.flow_runs.worker
Current Job Condition: {'type': 'Ready', 'state': 'CONDITION_RECONCILING', 'lastTransitionTime': '2025-11-18T17:24:08.227442Z'}
09:24:13 AM
prefect.flow_runs.worker
Submitting Cloud Run Job V2 berserk-baboon-793ca844a1b348d98c2013331a8710a4 for execution...
09:24:18 AM
prefect.flow_runs.worker
Cloud Run Job V2 berserk-baboon-793ca844a1b348d98c2013331a8710a4 submitted for execution with command: u v r u n - - n o - d e v p r e f e c t f l o w - r u n e x e c u t e
09:24:18 AM
prefect.flow_runs.worker
Completed submission of flow run '6a24cec0-11d2-417a-9531-ce9c32aedc38'
I have a cloudrun gcp setup as a worker pool but the run is hanging in pending state. I don't see anything in the worker logs except that the job was submitted. What else can I check to see what's wrong?Marvin
11/18/2025, 5:42 PMMarvin
11/18/2025, 5:52 PMberserk-baboon-...) -> Executions.
- Or via gcloud:
gcloud run jobs describe berserk-baboon-793ca844a1b348d98c2013331a8710a4 --region=<your-region>
gcloud run jobs executions list --job=berserk-baboon-793ca844a1b348d98c2013331a8710a4 --region=<your-region>
gcloud run jobs executions describe <execution-name> --region=<your-region>
- If there is no execution listed, the worker may lack permission to start executions (see IAM below).
2) Pull the Cloud Run execution logs
- In Logs Explorer, run a query for the job:
resource.type="cloud_run_job"
resource.labels.job_name="berserk-baboon-793ca844a1b348d98c2013331a8710a4"
- Or via gcloud:
gcloud logging read 'resource.type=cloud_run_job AND resource.labels.job_name="berserk-baboon-793ca844a1b348d98c2013331a8710a4"' --limit=200 --format=json
You’re looking for container startup errors like “Failed to pull image”, “uv: command not found”, missing env vars, Secret Manager access errors, VPC connector errors, etc.
3) Confirm the image and command are valid
- Prefect 3 Cloud Run worker uses the official image and uv by default:
- Image: prefecthq/prefect:3-latest
- Command shown spaced-out in logs is actually: uv run --no-dev prefect flow-run execute
- If you use a custom image, ensure it contains uv (or switch to the official Prefect 3 image). If uv is missing, the container will crash immediately and you’ll only see that in Cloud Run logs.
4) Check env vars made it into the job
- The container must have PREFECT_API_URL and PREFECT_API_KEY (or PREFECT_API_AUTH_STRING).
- Inspect the job’s container env:
gcloud run jobs describe berserk-baboon-793ca844a1b348d98c2013331a8710a4 --region=<your-region> --format=json | jq '.spec.template.template.containers[0].env'
- If you’re using env_from_secrets or prefect_api_key_secret, the job’s service account needs Secret Manager access (see IAM).
5) IAM: two service accounts often involved
- The identity that creates/submits the Job (your worker’s credentials) needs:
- roles/run.admin
- roles/iam.serviceAccountUser (to use the job’s execution SA if it’s different)
- The Job’s execution service account needs whatever the flow requires, commonly:
- Secret Manager access: roles/secretmanager.secretAccessor (if using env_from_secrets or prefect_api_key_secret)
- VPC/connector permissions if you set a VPC connector
- If no execution is created, it’s often a permission mismatch (run.admin or serviceAccountUser).
6) Networking/VPC connector
- If you set a vpc_connector_name but it’s misconfigured, startup can fail before logs reach Prefect.
- Validate connector exists and is Ready:
gcloud compute networks vpc-access connectors describe <connector-name> --region=<your-region>
7) Optional: keep the job for inspection
- In your work pool variables, set keep_job: true temporarily so the job remains and you can re-describe/inspect it after a failure.
8) Turn up worker logs (optional)
- If you’re running a local/hosted worker, start it with debug to see the exact job names and statuses:
PREFECT_LOGGING_LEVEL=DEBUG prefect worker start --pool <your-pool-name>
Common causes you’ll likely see in Cloud Run logs
- Custom image missing uv → Use prefecthq/prefect:3-latest or add uv.
- Missing Prefect env vars → Ensure PREFECT_API_URL + PREFECT_API_KEY are present.
- Secret Manager errors → Add roles/secretmanager.secretAccessor to the job’s service account.Marvin
11/18/2025, 5:52 PMenv_from_secrets or prefect_api_key_secret
I can pinpoint the fix quickly.