Great question — and sorry your worker is crashing. Assuming you’re on Prefect 3.x, here’s how to turn on verbose (DEBUG) logs for both your flow runs and your GCP Cloud Run Job worker so you can see what’s going on. How to get verbose logs 1) Flow run logs (show up in Prefect UI and via CLI) - Set

PREFECT_LOGGING_LEVEL=DEBUG

in the environment of the flow run. The easiest way is via deployment or work pool job variables: prefect.yaml (deployment-level)

deployments:
- name: my-deployment
  entrypoint: flows/my_flow.py:my_flow
  work_pool:
    name: my-cloud-run-pool
    job_variables:
      env:
        PREFECT_LOGGING_LEVEL: "DEBUG"

or in Python when deploying:

flow.deploy(
  name="my-deployment",
  work_pool_name="my-cloud-run-pool",
  job_variables={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG"}}
)

- View logs: - In the Prefect UI on the flow run page - Or via CLI:

prefect flow-run logs <flow-run-id> --tail

2) Worker logs (Cloud Run service logs) - Set

PREFECT_LOGGING_LEVEL=DEBUG

on your worker service so you get detailed polling/infra messages. If you deploy the worker with gcloud, something like:

gcloud run deploy prefect-worker \
  --image=prefecthq/prefect:3-latest \
  --set-env-vars \
    PREFECT_API_URL=$PREFECT_API_URL,\
    PREFECT_API_KEY=$PREFECT_API_KEY,\
    PREFECT_LOGGING_LEVEL=DEBUG \
  --service-account <YOUR_SA> \
  --args "prefect","worker","start","--pool","<WORK_POOL_NAME>","--with-healthcheck"

- Stream worker logs:

gcloud run logs read prefect-worker --follow --limit 200

3) If you need even more detail - For very low-level internals:

PREFECT_INTERNAL_LOGGING_LEVEL=DEBUG

- If you are self-hosting the API (Prefect Server), you can increase server logs with:

PREFECT_SERVER_LOGGING_LEVEL=DEBUG

Where logs go - Flow/task logs: Sent to Prefect Cloud and shown on the flow run page (also available via

prefect flow-run logs <id>

). They also print to stdout of the container. - Worker logs: Only in Cloud Run service logs (use

gcloud run logs read

). - Cloud Run Job execution logs: Your container’s stdout/stderr for the job execution will also show up in Cloud Run Job logs. Cloud Run-specific tips when the worker “crashes” - Check that the worker service can reach Prefect Cloud (correct

PREFECT_API_URL

and

PREFECT_API_KEY

, network egress allowed). - Make sure the worker service account has permissions to run Cloud Run Jobs and access Secret Manager if you use secrets. - Consider enabling the health check flag (

--with-healthcheck

) on the worker; Cloud Run can then probe

/health

. - Memory/CPU limits can trigger restarts — scan logs for OOMKilled or throttling. - You can add env vars to the job containers via work pool or deployment

job_variables.env

(e.g., to set

PREFECT_LOGGING_LEVEL=DEBUG

for the flow runs). Helpful commands you can use now

# Tail worker logs
gcloud run logs read prefect-worker --follow --limit 200

# Tail a flow run’s logs from Prefect Cloud
prefect flow-run logs <flow-run-id> --tail

Docs - Logging in Prefect 3.x: Logging - Settings and profiles: Settings & profiles - GCP Cloud Run worker guide: GCP worker guide - Troubleshooting Prefect Cloud: Cloud troubleshooting - Worker health checks: Worker health checks Quick questions to help pinpoint the crash - Is the crash happening in the worker service itself, or only when the worker launches a Cloud Run Job (flow run)? - Can you share the last 20–50 lines from

gcloud run logs read prefect-worker --tail

or the Cloud Run Job execution logs?

- Are you setting any

job_variables.env

or secrets for the flow that might be missing permissions?

Thanks for the detail. In Cloud Run, “Application exec likely failed” generally means the container could not start your process at all — i.e., the OS couldn’t exec the command. That happens before Prefect can emit any logs, so turning on Prefect DEBUG won’t show anything until we fix the container startup. Most common causes and how to check/fix: 1) Command/args formatting - Symptom: The entrypoint/command isn’t found or is passed as one big string. - Fix: Ensure the container can find

prefect

and that args are tokenized. With gcloud, use comma-separated `--args`:

gcloud run deploy prefect-worker \
    --image=prefecthq/prefect:3-latest \
    --set-env-vars=PREFECT_API_URL=$PREFECT_API_URL,PREFECT_API_KEY=$PREFECT_API_KEY,PREFECT_LOGGING_LEVEL=DEBUG \
    --service-account <YOUR_SA> \
    --args=prefect,worker,start,--pool,<WORK_POOL_NAME>,--with-healthcheck

If you set a custom command, make sure it is

prefect

and not quoted as a single string. 2) Wrong CPU architecture (exec format error) - Symptom: Image built on ARM (e.g., M1/M2) without amd64 support. Cloud Run needs amd64. - Fix: Rebuild multi-arch or amd64 image:

docker buildx build --platform linux/amd64 -t <your-image>:latest .

Check manifest:

docker buildx imagetools inspect <your-image>:latest

3) Missing Prefect binary or wrong PATH in custom image - Symptom:

exec: "prefect": executable file not found in $PATH

- Fix: Ensure Prefect is installed in the final image layer and on PATH (e.g.,

pip install prefect

or

pip install "prefect[gcp]"

), and you’re not activating a venv that doesn’t exist at runtime. - Quick sanity test locally:

docker run --rm \
    -e PREFECT_API_URL=$PREFECT_API_URL \
    -e PREFECT_API_KEY=$PREFECT_API_KEY \
    prefecthq/prefect:3-latest \
    prefect worker start --pool <WORK_POOL_NAME> --with-healthcheck

4) Shell script entrypoint issues - Symptom: Permission denied or “no such file” even though file exists. - Fix: If you wrap the command in a script, ensure: - Shebang present (e.g.,

#!/usr/bin/env bash

) - Unix line endings (LF), not Windows (CRLF) - Executable bits set:

chmod +x /path/to/script.sh

5) Wrong deployment target for the worker - The Prefect worker should run as a long-lived Cloud Run Service (so it can poll your work pool). Cloud Run Jobs are used by the worker to run your flow code. If you tried to run the worker itself as a Cloud Run Job, it won’t behave correctly. Deploy the worker as a Service and keep

--with-healthcheck

so it exposes

/health

on port 8080. 6) Health and port - For a Service, use

--with-healthcheck

so Cloud Run has a ready endpoint. Without it, Cloud Run may restart the container. This won’t usually cause “exec likely failed”, but it’s important for stability. What to do next - Share your exact deploy command (or Cloud Run container “Command” and “Args”) and image name/tag. That will help pinpoint if it’s args vs. image. - In GCP logs, look for specifics like “exec format error”, “executable file not found”, or “permission denied”. Those messages usually appear right before “Application exec likely failed”. - If you’re using a custom image, please share the relevant Dockerfile steps that install Prefect and set entrypoint/command. Once the container actually starts, enabling Prefect DEBUG will help: - Set for the worker:

PREFECT_LOGGING_LEVEL=DEBUG

- And for flow runs executed on Cloud Run Jobs (via job_variables on your work pool or deployment):

job_variables:
    env:
      PREFECT_LOGGING_LEVEL: "DEBUG"

References - Prefect GCP worker guide: GCP worker guide - Prefect logging in 3.x: Logging - Troubleshooting Prefect Cloud logs: Cloud troubleshooting

If you paste your gcloud deploy command (or a screenshot of the Cloud Run container configuration) and the last few lines from the Cloud Run logs, I can help you pinpoint the exact cause quickly.

Thanks for sharing the Dockerfile — that helps. The error “Application exec likely failed” almost always means the container could not start the command at all. In your case, the most suspicious bit is copying a prebuilt

.venv

from another stage and then prepending it to

PATH

. That can easily break the

prefect

executable at runtime and produce exactly this error. Why this often fails - The

prefect

CLI is a script with a shebang pointing to the Python inside the venv it was installed into. When you copy a venv from a different stage: - The shebang can point to a non-existent path in the runtime image. - Binary wheels compiled in the venv may not match the runtime base image or Python minor version. - Your custom venv can shadow the

prefect

that is already installed in the base

prefecthq/prefect

image. If your copied venv doesn’t include Prefect,

prefect

may not be found. - Result: Cloud Run tries to exec

prefect …

, but it’s either missing, not executable, or its shebang points to a Python that doesn’t exist → “Application exec likely failed”. Quick checks (run locally)

docker run --rm -it <your-image> sh -lc '
  echo "PATH=$PATH";
  command -v prefect || true;
  which python;
  python -V;
  if command -v prefect; then
    ls -l $(command -v prefect);
    head -1 $(command -v prefect);
  fi
'

If

command -v prefect

prints nothing, or the first line (shebang) points to an invalid path, that’s the issue. What I recommend 1) Don’t copy a prebuilt venv between stages. Reinstall your deps in the runtime image, so the Python, ABI, and paths match. - Minimal pattern:

FROM prefecthq/prefect:3-python3.13

     WORKDIR /opt/prefect/app
     COPY my_code ./my_code
     COPY requirements.txt .
     RUN pip install --no-cache-dir -r requirements.txt
     ENV PYTHONPATH="/opt/prefect/app"

- If you use pyproject/uv/poetry, install with the same Python in this runtime stage. 2) If you must use a venv, create it in the runtime stage (not copied from build) and make sure Prefect is installed into that venv:

FROM prefecthq/prefect:3-python3.13
   WORKDIR /opt/prefect/app

   RUN python -m venv /opt/prefect/venv
   ENV VIRTUAL_ENV=/opt/prefect/venv
   ENV PATH="$VIRTUAL_ENV/bin:$PATH"

   COPY requirements.txt .
   RUN pip install --no-cache-dir -r requirements.txt \
       && pip install --no-cache-dir prefect  # ensure prefect is in this venv

   COPY my_code ./my_code
   ENV PYTHONPATH="/opt/prefect/app"

3) Make sure you pass the command and args correctly to Cloud Run - Use comma-separated

--args

so each token is separate:

gcloud run deploy prefect-worker \
    --image=<your-image> \
    --set-env-vars=PREFECT_API_URL=$PREFECT_API_URL,PREFECT_API_KEY=$PREFECT_API_KEY,PREFECT_LOGGING_LEVEL=DEBUG \
    --service-account <YOUR_SA> \
    --args=prefect,worker,start,--pool,<WORK_POOL_NAME>,--with-healthcheck

- If you set the “Command” and “Args” fields in the UI, put

prefect

in Command and

worker,start,--pool,<WORK_POOL_NAME>,--with-healthcheck

as separate args (don’t paste the whole thing into Command as a single string). 4) Ensure architecture and Python match - Build for linux/amd64:

docker buildx build --platform linux/amd64 -t <your-image>:latest .

5) Once the container actually starts, enable verbose Prefect logs - Worker service:

PREFECT_LOGGING_LEVEL=DEBUG

- Flow runs via work pool job variables:

job_variables:
    env:
      PREFECT_LOGGING_LEVEL: "DEBUG"

If you can share: - Your full Dockerfile (including the “build” stage you’re copying from) - The exact

gcloud run deploy

command (or screenshot of “Command and args”) I can pinpoint the exact breakage. Most likely fix: stop copying

.venv

, install your deps into the runtime image, and keep

prefect

on PATH. References - GCP worker guide: GCP worker guide

- Logging in Prefect 3.x: Logging

Short answer: you can absolutely use uv, and you generally should not change the Prefect image’s ENTRYPOINT unless you have a specific reason. The official Prefect images already ship with uv and are designed to work with it. What the official image already gives you - uv is preinstalled and used to install Prefect and extras in the image. - ENTRYPOINT is a tiny+tini + entrypoint.sh that: - supports runtime installs via

EXTRA_PIP_PACKAGES

using uv - plays nicely with bash login shells and healthchecks - The image is built with UV_SYSTEM_PYTHON=1, UV_LINK_MODE=copy, and a uv cache for speed. Recommended patterns to leverage uv A) Base on the official Prefect image; install your deps with uv - Best for Prefect workers (Cloud Run or otherwise). Dockerfile:

FROM prefecthq/prefect:3-python3.13

WORKDIR /opt/prefect/app

# Install your workspace dependencies with uv (fast + cached)
# If you use requirements.txt:
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/uv \
    uv pip install --system -r requirements.txt

# Or if you use pyproject/uv.lock:
# COPY pyproject.toml uv.lock .
# RUN --mount=type=cache,target=/root/.cache/uv uv sync --frozen

# Copy your code
COPY my_code ./my_code

ENV PYTHONPATH=/opt/prefect/app
# Keep the default entrypoint
# ENTRYPOINT remains: /usr/bin/tini -g -- /opt/prefect/entrypoint.sh

- Then deploy your worker without touching the entrypoint:

gcloud run deploy prefect-worker \
    --image=<your-image> \
    --set-env-vars=PREFECT_API_URL=$PREFECT_API_URL,PREFECT_API_KEY=$PREFECT_API_KEY,PREFECT_LOGGING_LEVEL=DEBUG \
    --service-account <YOUR_SA> \
    --args=prefect,worker,start,--pool,<WORK_POOL_NAME>,--with-healthcheck

B) Use runtime extras via the provided entrypoint (optional) - If you want to keep the image minimal and add a couple of extras at runtime: - Set

EXTRA_PIP_PACKAGES

and the entrypoint will run:

uv pip install --system $EXTRA_PIP_PACKAGES

- Example (via work pool or Cloud Run env):

EXTRA_PIP_PACKAGES=prefect-gcp google-cloud-secret-manager

C) If you use uv-managed projects (pyproject.toml + uv.lock) - Build-time:

COPY pyproject.toml uv.lock .
  RUN --mount=type=cache,target=/root/.cache/uv uv sync --frozen
  COPY . .

- Runtime command can still be the plain Prefect CLI:

--args=prefect,worker,start,--pool,<WORK_POOL_NAME>,--with-healthcheck

You do not need to use

uv run

in the container if packages were installed with uv into the system Python. When to change the ENTRYPOINT - Only if you explicitly need to replace the entrypoint behavior (e.g., you don’t want the

EXTRA_PIP_PACKAGES

installer and bash-login semantics). - If you change it, keep tini and call Prefect directly:

ENTRYPOINT ["/usr/bin/tini", "-g", "--"]
  CMD ["prefect", "worker", "start", "--pool", "my-pool", "--with-healthcheck"]

- Or if you insist on

uv run

wrapping the command:

ENTRYPOINT ["/usr/bin/tini", "-g", "--", "uv", "run"]
  CMD ["prefect", "worker", "start", "--pool", "my-pool", "--with-healthcheck"]

Important gotchas (based on your earlier Dockerfile) - Avoid copying a prebuilt .venv from another stage. It often breaks shebangs and binaries and causes “Application exec likely failed”. Use uv to install into the system Python in the final image. - Ensure your image is linux/amd64 for Cloud Run: build with

--platform linux/amd64

. - Keep

prefect

on PATH. With the official image + uv system installs,

prefect

will be available. - For GCP features, install the right extras at build time:

RUN --mount=type=cache,target=/root/.cache/uv uv pip install --system "prefect[gcp]"

References - Official Prefect Dockerfile (shows uv + entrypoint): Prefect Dockerfile - Prefect GCP worker guide: GCP worker guide

- Logging in Prefect 3.x: Logging If you share your dependency layout (requirements.txt vs pyproject) I can suggest the most cache-efficient Dockerfile using uv.

Short answer: yes — install from uv.lock directly in the final image. The problem was copying a prebuilt

.venv

between stages. With uv, either install into the system Python in the final image or create the venv in the final image. No builder stage (for the venv) is needed. Two solid patterns: 1) Use uv.sync to a venv in the final image (keeps your lockfile intact) - Keep the default Prefect ENTRYPOINT. Just put the venv on PATH so

prefect

resolves from your uv-managed env. - Make sure Prefect (and any extras like

[gcp]

) is in your pyproject dependencies.

FROM prefecthq/prefect:3-python3.13

# Optional: uv performance knobs (uv already present in this image)
ENV UV_LINK_MODE=copy UV_COMPILE_BYTECODE=1

WORKDIR /opt/prefect/app

# Install exactly what’s in uv.lock into a local venv
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv uv sync --frozen --no-dev

# Ensure the venv is used at runtime
ENV VIRTUAL_ENV=/opt/prefect/app/.venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

# Your code
COPY . .
ENV PYTHONPATH=/opt/prefect/app

# Keep default ENTRYPOINT from the Prefect image (tini + entrypoint.sh)
# Deploy worker with:
# --args=prefect,worker,start,--pool,<WORK_POOL_NAME>,--with-healthcheck

2) Install into system Python with uv (no venv) - This avoids any PATH changes and works cleanly with the default ENTRYPOINT. It does not “use” uv.lock directly; if you must honor uv.lock, prefer pattern 1. If you’re okay with a regular PEP 517 install, use this:

FROM prefecthq/prefect:3-python3.13
ENV UV_SYSTEM_PYTHON=1 UV_LINK_MODE=copy UV_COMPILE_BYTECODE=1

WORKDIR /opt/prefect/app

# If your project can be installed as a package:
COPY pyproject.toml uv.lock ./
# Optional: if you want exact pins, stick with pattern 1 (uv sync); uv pip installs your project/build deps
RUN --mount=type=cache,target=/root/.cache/uv uv pip install --system .

COPY . .
ENV PYTHONPATH=/opt/prefect/app

# Keep default ENTRYPOINT

Should you update the ENTRYPOINT? - Not necessary in most cases. Keeping the default Prefect entrypoint is recommended because: - It supports runtime extras via

EXTRA_PIP_PACKAGES

using uv - It plays nicely with healthchecks and login-shell behavior If you prefer

uv run

instead of editing PATH: - You can swap ENTRYPOINT to

uv run

, but you’ll lose the runtime extras hook from the default entrypoint.

ENTRYPOINT ["/usr/bin/tini", "-g", "--", "uv", "run"]
CMD ["prefect", "worker", "start", "--pool", "my-pool", "--with-healthcheck"]

Cloud Run deploy reminder - Pass args as a comma-separated list (each token is one arg):

gcloud run deploy prefect-worker \
  --image=<your-image> \
  --set-env-vars=PREFECT_API_URL=$PREFECT_API_URL,PREFECT_API_KEY=$PREFECT_API_KEY,PREFECT_LOGGING_LEVEL=DEBUG \
  --service-account <YOUR_SA> \
  --args=prefect,worker,start,--pool,<WORK_POOL_NAME>,--with-healthcheck

Key points - Don’t copy a prebuilt

.venv

between stages — that’s what likely caused “Application exec likely failed.” - It’s perfectly fine to have a

.venv

in the final image; just ensure it’s on PATH (or use

uv run

). - If you use a

.venv

, include

prefect

(and extras like

prefect[gcp]

) in your project dependencies so the CLI is available inside the venv. - Ensure your base image Python version matches what’s in

uv.lock

(e.g., python = "3.13.*"). References: - Prefect Dockerfile (uv + entrypoint): Prefect Dockerfile - GCP worker guide: GCP worker guide

Short answer: In Prefect 3.x, if a Cloud Run worker pool’s job template has

command

unset, Prefect injects the default command and does not inherit your image’s CMD/ENTRYPOINT. Default command:

["prefect", "flow-run", "execute"]

Where this comes from: - Base worker default: prefect/workers/base.py (

_base_flow_run_command

returns

prefect flow-run execute

) - Applied by Cloud Run workers when preparing the job: - v2: prefect_gcp/workers/cloud_run_v2.py - v1: prefect_gcp/workers/cloud_run.py Tip: If you want to override it, set

command

in your worker pool’s job template (or job variables).

You can override it per-run by passing a

job_variables.command

string when you start the deployment run. Prefect will shlex-split the string into a list for Cloud Run. Options: - CLI (one-off run)

# By name
prefect deployment run "my-flow/my-deployment" \
  --job-variable 'command="python /workspace/custom_runner.py --flag value"'

# Or by id
prefect deployment run --id "<deployment-id>" \
  --job-variable 'command="bash -lc \"echo hello && prefect flow-run execute\""'

- Python API (one-off run)

from prefect.client import get_client
import asyncio

async def main():
    async with get_client() as client:
        await client.create_flow_run_from_deployment(
            name="my-flow/my-deployment",
            job_variables={"command": "python /workspace/custom_runner.py --flag value"},
        )

asyncio.run(main())

- Prefect UI - Open your deployment → Run → add a Job variable with key

command

and your command string. Notes: - Provide the command as a single string; Prefect converts it to

["arg0","arg1",...]

with shlex. - If your command includes spaces/quotes, wrap appropriately (e.g.,

bash -lc '...'

). - If you replace the default, ensure your command ultimately runs

prefect flow-run execute

unless you intentionally want a different entrypoint. Source for command handling (Prefect 3.x): - Base default command: prefect/workers/base.py - Cloud Run v2 worker command population: prefect_gcp/workers/cloud_run_v2.py - Cloud Run v1 worker command population: prefect_gcp/workers/cloud_run.py

BuildKit is enabled at the Docker daemon level, not via a Prefect flag. To use BuildKit when Prefect builds your image during deploy, set DOCKER_BUILDKIT=1 in the environment where you run prefect deploy (or enable it in the Docker daemon config). Quick options: - One-off in your shell

export DOCKER_BUILDKIT=1
prefect deploy -n my-deployment

- GitHub Actions / CI

- name: Deploy with BuildKit
  env:
    DOCKER_BUILDKIT: "1"
  run: prefect deploy -n my-deployment

- Python deploy

import os
os.environ["DOCKER_BUILDKIT"] = "1"

from prefect import flow
@flow
def my_flow(): ...
my_flow.deploy(
    name="my-deployment",
    work_pool_name="my-docker-pool",
    image="my-repo/my-image:tag",
)

- Permanent (daemon) Add to /etc/docker/daemon.json and restart Docker:

{
  "features": {
    "buildkit": true
  }
}

Notes: - Prefect 3.x uses docker-py to call the Docker build API; it does not have a dedicated BuildKit flag. If DOCKER_BUILDKIT is set or the daemon has BuildKit enabled, your deploy build will use it. - You can still pass normal docker build options in prefect.yaml via the docker build step (e.g., pull, platform, buildargs, nocache), but there’s no

buildkit

parameter. Relevant code paths: - Prefect deploy steps call the Docker build via the integrations step, which uses the Docker SDK; BuildKit selection happens in the daemon based on DOCKER_BUILDKIT/daemon settings.

It runs Prefect’s flow-run execution entrypoint. Specifically, `python -m prefect.engine`: - Reads the flow run ID from either the first CLI arg or

PREFECT__FLOW_RUN_ID

- Loads the flow run from the Prefect API and loads the flow code (via

PREFECT__FLOW_ENTRYPOINT

if set, otherwise from the deployment’s storage/source) - Executes the flow, handling engine signals (pause/abort) and exits with an appropriate code Examples:

# Pass the flow run id as an argument
python -m prefect.engine 12345678-1234-5678-1234-567812345678

# Or via env var
export PREFECT__FLOW_RUN_ID=12345678-1234-5678-1234-567812345678
python -m prefect.engine

Notes: - This is the “executor” for a single flow run; it’s not a worker. - Some workers (e.g., process-style) use this as their container command. Others (like Cloud Run) default to an equivalent

prefect flow-run execute

command. Source: - Engine entrypoint: prefect/engine.py

Use the venv’s Python (or Prefect CLI) directly from its bin directory. Equivalent options: - Using the engine module (what you have now, but from the venv)

command:
  - /opt/prefect/data-flows/.venv/bin/python
  - -m
  - prefect.engine

- Using the Prefect CLI (Prefect 3.x default behavior)

command:
  - /opt/prefect/data-flows/.venv/bin/prefect
  - flow-run
  - execute

Either will run the flow using the Prefect installed in

/opt/prefect/data-flows/.venv

. Make sure that path exists in your runtime image/environment.

Use the platform parameter on the build step. It’s passed straight through to the Docker SDK and supports values like linux/amd64 or linux/arm64. Example prefect.yaml:

build:
  - prefect_docker.deployments.steps.build_docker_image:
      requires: prefect-docker
      image_name: my-repo/my-image
      tag: latest
      dockerfile: Dockerfile
      platform: linux/amd64

Notes: - Format is os[/arch[/variant], e.g., linux/amd64, linux/arm64, linux/arm/v7. - This maps to docker-py’s build(platform=...) and ultimately instructs the daemon to build for that platform. - If you’re cross-building, ensure your Docker setup supports it (e.g., Buildx/QEMU).