<@ULVA73B9P> GCP Cloud Run worker pool fails to st...
# ask-marvin
m
@Marvin GCP Cloud Run worker pool fails to start and the only log I have is: terminated: Application failed to start: The container may have exited abnormally.
m
thought for 428.9 seconds
Sorry you’re hitting that — Cloud Run’s “Application failed to start” usually means the container never bound to a port or exited before a health probe could succeed. Let’s triage quickly. First, a couple clarifying questions: - Are you running a hybrid worker (Cloud Run service that polls a work pool) or using a push work pool (no worker to run)? - What image and startup command/args are you using? (from the Cloud Run service’s Revision details) - Which env vars did you set (especially
PREFECT_API_URL
,
PREFECT_API_KEY
)? - Any Cloud Logging entries under the service’s latest revision? If you’re using a hybrid worker on Cloud Run, here’s a known-good baseline to get it to start and stay healthy: 1) Required env vars (assuming Prefect 3.x/Cloud)
Copy code
PREFECT_API_URL=<https://api.prefect.cloud/api/accounts/<ACCOUNT_ID>/workspaces/<WORKSPACE_ID>>
PREFECT_API_KEY=<YOUR_API_KEY>
# Ensure the health server binds correctly; Cloud Run sets $PORT
PREFECT_WORKER_WEBSERVER_HOST=0.0.0.0
PREFECT_WORKER_WEBSERVER_PORT=8080   # or set to $PORT explicitly
PREFECT_LOGGING_LEVEL=DEBUG          # for visibility while debugging
2) Deploy the worker service with a stable command and healthcheck
Copy code
gcloud run deploy prefect-worker \
  --image=prefecthq/prefect:3-latest \
  --set-env-vars PREFECT_API_URL=$PREFECT_API_URL,PREFECT_API_KEY=$PREFECT_API_KEY,PREFECT_WORKER_WEBSERVER_HOST=0.0.0.0,PREFECT_WORKER_WEBSERVER_PORT=8080,PREFECT_LOGGING_LEVEL=DEBUG \
  --service-account <YOUR-SA>@<PROJECT>.<http://iam.gserviceaccount.com|iam.gserviceaccount.com> \
  --no-cpu-throttling \
  --min-instances 1 \
  --cpu 1 --memory 1Gi --timeout 900 \
  --region <REGION> \
  --startup-probe httpGet.port=8080,httpGet.path=/health,initialDelaySeconds=180,periodSeconds=20,timeoutSeconds=20 \
  --args "prefect","worker","start","--with-healthcheck","-p","<YOUR_WORK_POOL_NAME>"
Notes: -
--with-healthcheck
starts the worker’s HTTP health server at
/health
(host
0.0.0.0
, port
8080
by default). - Make sure the Cloud Run startup probe points to the same port/path. - Worker type is inferred from the work pool; you do not need to pass a type flag to
prefect worker start
. Common root causes of this exact Cloud Run error and quick checks: - Missing Prefect API env vars - Verify in the revision’s configuration that both
PREFECT_API_URL
and
PREFECT_API_KEY
are set correctly. - Nothing listening on the expected port - Ensure
--with-healthcheck
is present and
PREFECT_WORKER_WEBSERVER_HOST=0.0.0.0
,
PREFECT_WORKER_WEBSERVER_PORT=8080
. - CPU throttling or too little memory during startup - Start with
--memory 1Gi
and
--no-cpu-throttling
. - Startup probe too aggressive - Increase
initialDelaySeconds
(e.g., 180s) so the worker has time to initialize. - Wrong image architecture - If you built a custom image on Apple Silicon, rebuild for
linux/amd64
. - Incorrect command/args - In Cloud Run, confirm container args match:
prefect worker start --with-healthcheck -p <pool>
. - Network egress/VPC issues (rare for startup) - If using a VPC connector with restricted egress, the worker may not reach Prefect API. How to get more visibility right now: - Stream service logs
Copy code
gcloud run logs read prefect-worker --region <REGION> --limit 200 --follow
- Confirm the args and env on the deployed revision
Copy code
gcloud run services describe prefect-worker --region <REGION> \
  --format="value(spec.template.spec.containers[0].args)"
gcloud run services describe prefect-worker --region <REGION> \
  --format="value(spec.template.spec.containers[0].env)"
- If the worker comes up, the
/health
endpoint should return 200 OK. You can curl it if the service is publicly accessible. If you don’t actually need to run a long-lived worker, consider using a push work pool (no worker, Prefect Cloud pushes to Cloud Run directly). It’s simpler and avoids this class of startup issues:
- Guide: Serverless push work pools (Cloud Run) Helpful docs and references: - GCP Cloud Run worker guide - Worker healthchecks (defaults: host 0.0.0.0, port 8080, path /health) - Cloud Run provisioner reference If you can share: - The work pool type/name - The Cloud Run service’s image + args - The env var list from the revision - Region and service account roles …I can pinpoint the exact misconfiguration and provide a tailored fix.
m
@Marvin how about this error:
Copy code
Flow run could not be submitted to infrastructure: Parameter "parent" value "projects/custom_project/locations/" does not match the pattern "^projects/[^/]+/locations/[^/]+$"
m
thought for 117.8 seconds
That error means the Cloud Run API call was built with an empty region in the “parent” path. The API expects: - projects/{PROJECT_ID}/locations/{REGION} …but yours looks like: - projects/custom_project/locations/ ← missing region Where to fix it depends on whether you’re using a Cloud Run worker (hybrid) or a Cloud Run push work pool: What to check - Work pool type: Is it a Cloud Run v2 worker pool or a push pool? - Region field: - In either case, Prefect builds the “parent” value from: - project: taken from the GCP Credentials block you attach to the work pool/template - region: taken from the work pool’s base job template’s region variable - GCP Credentials block: Ensure it contains a valid project id (e.g., my-project-123). If it’s blank or points to the wrong project, fix it there. Exact fields that control this - For Cloud Run v2 worker and push pools, Prefect uses the same underlying config: - credentials: a GCP Credentials block; its project value populates the {PROJECT_ID} - region: a string like us-central1, us-east1, europe-west1; this populates the {REGION} - Prefect constructs the parent string at runtime as: - projects/{credentials.project}/locations/{region} Typical causes of your error - Region missing in the work pool’s base job template - The “region” field is empty or unset - Credentials block missing project - The block has no project set, or it’s set to a placeholder - Mismatch between deployment and pool overrides - A deployment override unexpectedly sets region to an empty string How to fix - Open your Cloud Run work pool in the Prefect UI and set: - GCP Credentials: pick the correct credentials block with a valid project id - Region: set to a valid region, e.g., us-central1 - If you’re using CLI/JSON for the base job template, ensure it includes:
Copy code
{
  "variables": {
    "credentials": "blocks/gcp-credentials/<your-block-name>",
    "region": "us-central1"
  }
}
- If you override in a deployment, verify the deployment’s infrastructure overrides don’t clear region. Quick verification - Run a small test flow against the same work pool and watch the logs; if set correctly, the error disappears. - If you can, paste the pool’s base job template snippet (with secrets redacted) here and I’ll spot-check the region and credentials fields. References - Cloud Run worker guide: GCP Cloud Run worker guide - Serverless push pools (Cloud Run): Serverless push work pools - Worker healthchecks (unrelated to this error, but useful): Worker healthchecks