Short answer: it’s trying to create that directory inside the container where the process is running. Creating a directory on the host will not help unless you mount it into the container at that exact path. What’s happening - Prefect uses

PREFECT_HOME

for its local state. By default it’s

~/.prefect

relative to the user inside the process. In Docker, that’s inside the container (often

/root/.prefect

unless you run as a different user). - Your warning shows

Failed to create the Prefect home directory at /home/user/prefect-d

, which means

PREFECT_HOME

is set to that path inside the process. In most Prefect Docker images there is no

/home/user

and the container user may not have permission to create it, so the mkdir fails. How to check what’s set - If the warning is in the worker logs, exec into the worker container and run:

prefect config view --show-sources | grep PREFECT_HOME -A2

- If it’s in a flow-run container, inspect the job variables/environment passed by the Docker work pool/deployment. Ways to fix - Simplest: do not set

PREFECT_HOME

and let it default to a writable path inside the container (typically

/root/.prefect

). - Or set

PREFECT_HOME

to a known-writable path that exists in the container (e.g.,

/opt/prefect

) and, if you want persistence, mount a volume there. Examples 1) Worker container (persist home):

docker run --name prefect-worker \
  -e PREFECT_API_URL=<your-api-url> \
  -e PREFECT_HOME=/opt/prefect \
  -v prefect-worker-home:/opt/prefect \
  prefecthq/prefect:3-latest \
  prefect worker start -p <your-docker-pool>

2) Flow-run containers (via job variables on the work pool or deployment): - Set

env.PREFECT_HOME

and mount a volume to match:

job_variables:
  env:
    PREFECT_HOME: /opt/prefect
  volumes:
    - type: bind
      source: /var/lib/prefect/flow-home
      target: /opt/prefect
      read_only: false

Notes - If you’re running as a non-root user in the image, ensure the target directory exists and is writable (or create it in your Dockerfile and chown it). - Creating

.prefect

on the host won’t help unless you mount it to the same path inside the container. Helpful docs - Settings and profiles: Settings and profiles - Managing settings (including

PREFECT_HOME

): Manage settings - Customizing Docker job variables (env, volumes, image, networks): Customize job variables To help further, can you share: - Where you see the warning (worker logs vs flow-run logs)? - Your worker start command or docker-compose snippet - Any

job_variables

(env/volumes) on your work pool or deployment, especially anything setting

PREFECT_HOME

?

Got it — if you’re seeing zero logs in the UI and nothing from

docker logs

on your worker container, it usually means the worker either isn’t actually starting, isn’t connected to your API, or can’t create job containers. Here’s a quick checklist to pinpoint the issue: 1) Confirm you’re tailing the right container - Are you watching the worker container (the one that runs

prefect worker start ...

) vs. a flow-run job container? If the worker never connects, no job container will ever be created. 2) Verify the worker is actually running - Check it’s Up:

docker ps
docker logs <worker-container>

You should see logs like “Starting worker” and “Polling work pool …”. If you see nothing at all, the command/entrypoint may not be launching the worker. 3) Ensure the worker can reach the Prefect API - For Prefect Cloud: set both

PREFECT_API_URL

and

PREFECT_API_KEY

. - For Prefect Server: set

PREFECT_API_URL=http://<server-host>:4200/api

. - Inside the container, verify:

prefect config view --show-sources | grep -E 'PREFECT_API_URL|API_KEY' -A2

If the API isn’t configured, the worker will fail immediately and won’t show up in the UI. 4) Make sure the worker is polling the correct pool/queue - Start with:

prefect worker start -p <your-docker-pool>

- Confirm your deployment targets the same pool (and queue, if used). Otherwise, your runs will never be claimed. 5) If your worker runs inside Docker and launches Docker jobs, mount the Docker socket - Without the socket, the worker can’t create flow-run containers:

-v /var/run/docker.sock:/var/run/docker.sock

6) Enable more logging to see early failures - Set:

-e PREFECT_LOGGING_LEVEL=DEBUG

A “known-good” worker run example - Prefect Server:

docker run --name prefect-docker-worker \
  -e PREFECT_API_URL=http://<server-host>:4200/api \
  -e PREFECT_LOGGING_LEVEL=DEBUG \
  -v /var/run/docker.sock:/var/run/docker.sock \
  prefecthq/prefect:3-latest \
  prefect worker start -p <your-docker-pool>

- Prefect Cloud:

docker run --name prefect-docker-worker \
  -e PREFECT_API_URL=<https://api.prefect.cloud/api/accounts/<account_id>/workspaces/<workspace_id>> \
  -e PREFECT_API_KEY=<your_key> \
  -e PREFECT_LOGGING_LEVEL=DEBUG \
  -v /var/run/docker.sock:/var/run/docker.sock \
  prefecthq/prefect:3-latest \
  prefect worker start -p <your-docker-pool>

If the worker starts correctly, you’ll also see it appear as “online” under Work Pools in the UI. If this doesn’t resolve it, can you share: - The exact

docker run

or docker-compose service for your worker - Whether you’re using Prefect Cloud or Prefect Server (and your

PREFECT_API_URL

) - The work pool name and type (should be “docker”) - Whether the Docker socket is mounted Helpful docs: - Settings and profiles: Settings and profiles - Manage settings (including API config): Manage settings - Customizing Docker job variables (env, image, volumes, networks): Customize job variables

Thanks for the details. A couple of quick clarifiers so we don’t chase the wrong thing: - Which container is exiting immediately when you start a flow: the worker container itself, or the flow-run job container that the worker launches? - If it’s the worker container, what’s its exit code (

docker inspect <worker> --format '{{.State.ExitCode}}'

)? - If it’s the job container, what image are you using and do you see any container at all show up when a run is submitted (

docker ps -a

right after you click “Run”)? In parallel, here are the most common causes and fixes for “container exits immediately with no logs”: 1) Worker container exits or crashes - API not configured: Ensure the worker container has API env set. For Cloud:

PREFECT_API_URL

+

PREFECT_API_KEY

. For Server:

PREFECT_API_URL=http://<server-host>:4200/api

. - Remove the

--type docker

flag. The worker type is derived from the pool; this flag isn’t needed and can cause confusion. Use:

prefect worker start -p dock-pool --name d1

- Make sure the process can write its home dir. Earlier you hit a “Failed to create Prefect home directory” warning; inside the worker container set a safe path and mount a volume:

-e PREFECT_HOME=/opt/prefect
-v prefect-worker-home:/opt/prefect

- Turn on debug logs to catch early failures:

-e PREFECT_LOGGING_LEVEL=DEBUG

- Full known-good worker run (also mounts Docker socket so it can launch job containers):

docker run --name prefect-docker-worker \
  -e PREFECT_API_URL=<your-api-url> \
  -e PREFECT_API_KEY=<only for Cloud> \
  -e PREFECT_HOME=/opt/prefect \
  -e PREFECT_LOGGING_LEVEL=DEBUG \
  -v prefect-worker-home:/opt/prefect \
  -v /var/run/docker.sock:/var/run/docker.sock \
  prefecthq/prefect:3-latest \
  prefect worker start -p dock-pool --name d1

2) Job container exits immediately (worker stays up) - Image missing Prefect or Python: Use an image with Prefect 3 installed (e.g.,

prefecthq/prefect:3-python3.11

or your own image with

pip install prefect

+ your deps). Set it via job variables at the pool or deployment level. - Overridden entrypoint/command: If your job variables override

entrypoint

or

command

, the container may do nothing and exit. Remove overrides unless you know you need them. - No Docker access: If the worker can’t create job containers (missing Docker socket mount), you’ll see failures in the worker logs. Mount

/var/run/docker.sock:/var/run/docker.sock

in the worker. - Source access fails: If your deployment pulls code with

.from_source(...)

, ensure the job container has network access and any credentials needed to pull your repo. A quick test is to deploy with a simple image and a public repo first. Example deployment specifying a safe image and debug env:

flow.from_source(
    "<https://github.com/<org>/<repo>>",
    "flows/my_flow.py:flow"
).deploy(
    name="my-deploy",
    work_pool_name="dock-pool",
    job_variables={
        "image": "prefecthq/prefect:3-python3.11",
        "env": {"PREFECT_LOGGING_LEVEL": "DEBUG"},
    },
)

Quick checks - Confirm the worker is online in the UI under Work Pools after it starts. - Tail worker logs and watch what happens when you trigger a run:

docker logs -f prefect-docker-worker

- If the worker exits when a run is submitted, capture the exit code:

docker inspect prefect-docker-worker --format '{{.State.ExitCode}} {{.State.OOMKilled}}'

- If a job container is created and exits, list it right after you click run:

docker ps -a --no-trunc | grep -i prefect

Docs for reference - Settings and profiles (API, logging, PREFECT_HOME): Settings and profiles - Manage settings: Manage settings - Customize Docker job variables (image, env, volumes, networks): Customize job variables

If you can share: - The exact

docker run

or compose for your worker - Whether the worker or job container is exiting - The job

image

you’re using and any

job_variables

(env/volumes/entrypoint/command) I can pinpoint the next step quickly.

✅ thought for 46.8 seconds

Thanks — since it’s the job container that exits immediately, the two most common culprits are: 1) The image overrides Prefect’s command via ENTRYPOINT - Your Dockerfile sets:

ENTRYPOINT ["python", "aubian_extractor_flow.py"]

- For Docker workers, Prefect injects its own command to start the flow run. A custom ENTRYPOINT typically overrides that, so your container runs your script instead of Prefect’s engine and then exits. - Fix: remove the ENTRYPOINT from the image so Prefect can set the run command. Rebuild and redeploy. 2) Prefect is not installed in the job image - If the image doesn’t have Prefect 3 installed, the default Prefect command will fail immediately and the container will exit before any logs get emitted. - Quick test:

docker run --rm --entrypoint python <your-image> -c "import prefect; print(prefect.__version__)"

- If that fails, add Prefect to your requirements and rebuild (e.g.,

prefect>=3,<4

). Recommended path to confirm - Temporarily switch your deployment’s job image to a known-good Prefect image to validate everything else:

job_variables:
  image: prefecthq/prefect:3-python3.11
  env:
    PREFECT_LOGGING_LEVEL: DEBUG

If runs work with this image, the issue is with your custom image. What to change in your Dockerfile - Remove the ENTRYPOINT line. - Ensure Prefect is installed in the final stage (either included in requirements.txt or explicitly installed). For example:

# In builder stage packages (requirements.txt) include: prefect>=3,<4
# Or explicitly:
RUN pip install --no-cache-dir "prefect>=3,<4"

- Keep your code and dependencies as you have them, but don’t override the runtime command. Extra debugging tips - See exactly what the Docker worker plans to run:

prefect work-pool get-default-base-job-template -t docker

- Add debug logging for the job:

job_variables:
  env:
    PREFECT_LOGGING_LEVEL: DEBUG

Docs - Customize Docker job variables: Customize job variables If you can share your deployment config (how you set

image

, any `entrypoint`/`command` overrides in job variables, and whether your requirements include Prefect), I can pinpoint the exact change.