Paulien Jeunesse
08/08/2025, 1:33 PMMarvin
08/08/2025, 1:34 PMMarvin
08/08/2025, 1:44 PMPaulien Jeunesse
08/08/2025, 1:47 PMMarvin
08/08/2025, 1:48 PMMarvin
08/08/2025, 1:50 PMPaulien Jeunesse
08/15/2025, 2:02 PMMarvin
08/15/2025, 2:02 PMMarvin
08/15/2025, 2:15 PMprefect deploy
.
- Verify Docker is up: docker info
(or start Docker Desktop; on Linux: sudo systemctl start docker
).
If you want Docker
1) Ensure Docker is installed and running.
2) Make sure the Docker extra is available:
pip install "prefect[docker]"
3) Use a Docker work pool and a running worker:
prefect work-pool create --type docker my-docker-pool
prefect worker start --pool my-docker-pool
4) Point your deployment at that pool and (if you’re building/pulling an image) re-run once Docker is up.
If you don’t want Docker
- Create and point the deployment to a Process pool instead:
prefect work-pool create --type process my-process-pool
- Update your deployment to use my-process-pool
so no Docker is involved.
A little more context never hurts:
- What OS are you on?
- What’s your work pool type and name?
- Are you building an image in prefect deploy
, or just enabling in the UI?
References (such as they are):
- Prefect 3 deployments API
- Prefect 3.2 release notes
- Work pools and workers overview
- Prefect + Docker overview
Share your deployment config or the exact command you ran and I’ll point you at the precise fix instead of this thrilling diagnostic checklist.Paulien Jeunesse
08/15/2025, 2:32 PMMarvin
08/15/2025, 2:32 PMMarvin
08/15/2025, 2:34 PMFlow.from_source(...)
so the worker can pull your code (e.g., from Git).
- Make sure the image has git and your dependencies installed.
from prefect import Flow
flow_from_git = Flow.from_source(
source="<https://github.com/your-org/your-repo>",
entrypoint="path/to/file.py:your_flow",
)
await flow_from_git.deploy(
name=new_deployment_name,
work_pool_name=self.work_pool_name, # my-podman-pool
image="your-image-with-git-and-deps",
build=False,
push=False,
)
2) Bake your code into the image (no pulling at runtime)
- Build your own image that includes your code and deps, then point the deployment at it.
- With Podman on the same host as the worker, push=False
is fine.
# Dockerfile (works with Podman too)
FROM prefecthq/prefect:3.2.15-python3.12
WORKDIR /opt/prefect/flows
COPY . /opt/prefect/flows
# Optional: if you have deps
# COPY requirements.txt .
# RUN pip install -r requirements.txt
Build and deploy:
podman build -t local/my-flow:dev .
# or docker build ...
await flow.deploy(
name=new_deployment_name,
work_pool_name=self.work_pool_name,
image="local/my-flow:dev",
build=False,
push=False,
)
3) Mount your local code into the container (dev-only convenience)
- Tell the worker to bind-mount your source into the container via job_variables
.
- For Podman on SELinux systems, add :Z
to relabel.
from pathlib import Path
await flow.deploy(
name=new_deployment_name,
work_pool_name=self.work_pool_name,
image="prefecthq/prefect:3.2.15-python3.12",
build=False,
push=False,
job_variables={
"volumes": [f"{Path.cwd()}:/opt/prefect/flows:Z"], # drop :Z if not needed
"env": {"PYTHONPATH": "/opt/prefect/flows"},
},
)
Notes:
- This only works if the worker host can see your local path (i.e., same machine).
- Your deployment’s entrypoint must be under the mounted path.
Which path do you prefer: pulling from a repo, baking an image, or mounting locally? If you share your repo layout and where your worker runs, I’ll give you the exact incantation instead of this choose-your-own-adventure.Paulien Jeunesse
08/15/2025, 4:05 PMMarvin
08/15/2025, 4:05 PMMarvin
08/15/2025, 4:06 PM/opt/prefect
as its working directory, so it will look for your entrypoint under /opt/prefect/...
. Your WORKDIR /opt/prefect/flows
in the Dockerfile gets overridden at runtime. Thrilling, I know.
You’ve got three clean fixes (pick one):
A) Put your project under /opt/prefect (recommended if your entrypoint is “src/…”)
- Update your Dockerfile to match the worker’s expectations:
FROM prefecthq/prefect:3.2.15-python3.12
WORKDIR /opt/prefect
COPY . /opt/prefect
RUN pip install -e . --find-links ./local_pypi/
ENV PYTHONPATH="/opt/prefect/src:$PYTHONPATH"
- Use an entrypoint like src/path/to/flow.py:flow
. And point your deployment to this custom image:
await flow.deploy(
name=new_deployment_name,
work_pool_name=self.work_pool_name,
image="local/my-flow:dev",
build=False,
push=False,
)
B) Keep your code under /opt/prefect/flows, but change the container’s working dir
- Tell the worker to run in /opt/prefect/flows
so the entrypoint resolves there:
await flow.deploy(
name=new_deployment_name,
work_pool_name=self.work_pool_name,
image="prefecthq/prefect:3.2.15-python3.12",
build=False,
push=False,
job_variables={"working_dir": "/opt/prefect/flows"},
)
- Then make sure your deployment entrypoint is relative to /opt/prefect/flows
.
C) Keep the symlink
- Your symlink from /opt/prefect/flows/src
to /opt/prefect/src
will work. It’s just… a bit hacky. If it keeps you moving, fine.
Important notes:
- Even if you pip install -e .
, Prefect still needs to open the entrypoint file path inside the container. That’s why the file must exist under the container’s working dir (or you adjust the working dir).
- Don’t forget to point the deployment at your custom image if you’re baking code into it. Using the base prefecthq/prefect
image without a volume mount means no code inside the container.
If you share your deployment entrypoint string and where that file sits in the image, I’ll give you the exact minimal change.Paulien Jeunesse
08/15/2025, 4:24 PMMarvin
08/15/2025, 4:24 PMMarvin
08/15/2025, 4:25 PMIMAGE_TAG=$(git rev-parse --short HEAD)
podman build -t local/huscarl:${IMAGE_TAG} .
await flow.deploy(
name=new_deployment_name,
work_pool_name=self.work_pool_name,
image=f"local/huscarl:{IMAGE_TAG}",
build=False,
push=False,
)
- You don’t need to disable/stop the deployment; updating it is enough. Existing running flow runs finish on the old image; future runs use the new one.
To speed rebuilds, structure your Dockerfile to maximize cache:
FROM prefecthq/prefect:3.2.15-python3.12
WORKDIR /opt/prefect
# Copy only dependency files first so this layer caches
COPY pyproject.toml poetry.lock* requirements.txt* /opt/prefect/
RUN pip install -r requirements.txt # or poetry install --no-root
# Now copy the source
COPY . /opt/prefect
RUN pip install -e . --find-links ./local_pypi/
ENV PYTHONPATH="/opt/prefect/src:$PYTHONPATH"
2) Fetch code at runtime (no rebuilds)
- Use Flow.from_source(...)
pointing at your Git repo. Push code changes; the worker clones the repo each run. Your base image just needs git + deps.
from prefect import Flow
flow = Flow.from_source(
source="<https://github.com/your-org/huscarl>",
entrypoint="src/huscarl/flows/my_flow.py:flow",
)
await flow.deploy(
name=new_deployment_name,
work_pool_name=self.work_pool_name,
image="prefecthq/prefect:3.2.15-python3.12", # or your base with git+deps
build=False,
push=False,
)
3) Bind-mount your code (dev-only)
- Mount your local source into the container so changes appear instantly; no rebuild, but only works if the worker runs on the same host.
await flow.deploy(
name=new_deployment_name,
work_pool_name=self.work_pool_name,
image="prefecthq/prefect:3.2.15-python3.12",
build=False,
push=False,
job_variables={
"volumes": ["/path/to/your/repo:/opt/prefect:Z"], # :Z for SELinux/Podman
"working_dir": "/opt/prefect",
"env": {"PYTHONPATH": "/opt/prefect/src"},
},
)
TL;DR
- Baking code into images = rebuild per change; just update the deployment, don’t stop it.
- Want to avoid rebuilds? Use from_source
or a volume mount during development.
Tell me which path you prefer and I’ll tailor the exact commands for your Podman setup.Paulien Jeunesse
08/15/2025, 4:31 PMMarvin
08/15/2025, 4:31 PMMarvin
08/15/2025, 4:32 PMpodman run -d --name prefect-postgres -e POSTGRES_USER=prefect -e POSTGRES_PASSWORD=prefect -e POSTGRES_DB=prefect -p 5432:5432 postgres:16
- Point Prefect Server at it and restart the server:
export PREFECT_API_DATABASE_CONNECTION_URL="<postgresql+asyncpg://prefect:prefect@localhost:5432/prefect>"
prefect server start
Quick mitigations (if you must stay on SQLite)
- Ensure only one prefect server start
process is running against the DB.
- Run fewer workers/concurrency for now (heartbeats are frequent writes).
- Keep the DB on a local filesystem (not NFS/remote). If you run the server in a container with a bind mount on SELinux, use :Z
on the volume.
- Bump SQLite timeout to reduce “database is locked” errors (still a band-aid):
export PREFECT_API_DATABASE_CONNECTION_URL="sqlite+aiosqlite:///$(prefect config view | grep DEFAULT_PREFECT_HOME -A1 | tail -1 | awk '{print $2}')/prefect.db?timeout=60"
prefect server start
(or set the path directly: sqlite+aiosqlite:////home/you/.prefect/prefect.db?timeout=60
)
A couple of confirmations that will help me tailor the exact steps:
- Are you running Prefect Server locally with the default SQLite?
- How many workers are heartbeating to this server?
- Is the server running in a container with a bind-mounted DB path?Paulien Jeunesse
08/15/2025, 5:06 PMMarvin
08/15/2025, 5:07 PMMarvin
08/15/2025, 5:07 PMfile+.vscode-resource... | /root/.prefect/storage/...
path is a dead giveaway: the worker can’t see that filesystem, so it can’t load your flow.
Pick one of these fixes:
1) Bake code into the image (what you were doing)
- Ensure your deployment actually uses your custom image and the entrypoint path matches where the code lives in that image (e.g., /opt/prefect/src/...
).
- Rebuild the image when code changes, redeploy pointing at the new tag.
2) Fetch code at runtime (no image rebuilds)
- Use Flow.from_source(...)
so the worker clones your repo at run time.
from prefect import Flow
flow = Flow.from_source(
source="<https://github.com/your-org/your-repo>",
entrypoint="src/your_pkg/flows/your_flow.py:flow",
)
await flow.deploy(
name="your-deployment",
work_pool_name="my-podman-pool",
image="prefecthq/prefect:3.2.15-python3.12", # ensure git + deps are available
build=False,
push=False,
)
3) Dev-only: mount your code into the worker container
- If the worker runs on the same host, bind-mount your project and set working_dir
so the entrypoint resolves.
await flow.deploy(
name="your-deployment",
work_pool_name="my-podman-pool",
image="prefecthq/prefect:3.2.15-python3.12",
build=False,
push=False,
job_variables={
"volumes": ["/absolute/path/to/repo:/opt/prefect:Z"], # drop :Z if not on SELinux/Podman
"working_dir": "/opt/prefect",
"env": {"PYTHONPATH": "/opt/prefect/src"},
},
)
Why this happened
- With build/push disabled and no from_source
, Prefect recorded a file-based storage URL (under ~/.prefect/storage/...
) on the machine that executed deploy
. Your worker container can’t access that path, especially when it came from a VS Code devcontainer.
If you share:
- where your worker runs (host/container),
- your deployment’s image and entrypoint,
- and whether you want image-based vs repo-based delivery,
I’ll give you the minimal change to stop Prefect from chasing VS Code’s ghost paths.Paulien Jeunesse
08/15/2025, 5:10 PMMarvin
08/15/2025, 5:11 PMMarvin
08/15/2025, 5:12 PMpip install "prefect[aws]"
- Provide cloud creds in the worker environment (AWS/GCP/Azure).
- Configure default result storage to your bucket/prefix and enable persistence. In 3.x this is done via Prefect settings or per-task/flow config. See: Prefect 3: Results (persistence and storage)
- Pass the relevant env via your deployment’s job_variables.env
so the flow-run container sees them.
If you want, tell me which backend (S3, GCS, Azure) and I’ll give you the exact env/config to drop into your deployment.
Option B: Stick with local filesystem results (dev-only or on single host)
- Bind-mount a host directory into the flow-run containers at the path Prefect uses so it actually exists and persists across runs.
- For your setup (root inside container), mount /root/.prefect
or just the storage
subdir:
await flow.deploy(
name="your-deployment",
work_pool_name="my-podman-pool",
image="your-custom-image",
build=False,
push=False,
job_variables={
# Change the left-hand path to a real directory on the worker host
"volumes": ["/srv/prefect-results:/root/.prefect:Z"], # :Z if SELinux/Podman
"env": {"PREFECT_RESULTS_PERSIST_BY_DEFAULT": "true"},
},
)
- Make sure the entrypoint still points to the code baked into the image. This only fixes the results location.
- Caveat: the UI’s clickable path still won’t be readable from your laptop; the files live on the worker host. This is why remote object storage is nicer.
Why this happens
- With persist_result=True
(globally or on tasks), and no explicit remote storage, Prefect defaults to a local path like /root/.prefect/storage/...
inside the container. The directory may not exist, and even if it does, nothing outside that container can read it. The VS Code URL in the error is just the UI trying to be helpful.
Tell me:
- Which backend you prefer for result storage (S3/GCS/Azure/local), and
- Whether you read results only via the Prefect UI/API or also directly from files.
I’ll give you the exact config (env vars and deployment job_variables
) for your choice.Paulien Jeunesse
08/15/2025, 5:14 PMMarvin
08/15/2025, 5:14 PMMarvin
08/15/2025, 5:16 PM# Pick a host dir for results and Prefect home
mkdir -p /srv/prefect-home
# In your deployment call:
await flow.deploy(
name="huscarl-dev",
work_pool_name="my-podman-pool",
image="your-custom-image",
build=False,
push=False,
job_variables={
"volumes": [
"/srv/prefect-home:/opt/prefect-home:Z" # :Z for SELinux/Podman
],
"env": {
"PREFECT_HOME": "/opt/prefect-home",
"PREFECT_RESULTS_PERSIST_BY_DEFAULT": "true",
# optional: steer results into a predictable subdir
"PREFECT_RESULTS_DEFAULT_STORAGE": "/opt/prefect-home/storage",
},
},
)
Notes:
- Mounting PREFECT_HOME
ensures Prefect’s storage (including result storage) lives on the host path.
- The UI may link to filesystem paths; they won’t be clickable from your laptop, but the files will exist on the host.
- You can also set persist_result=True
on your flow/tasks:
from prefect import flow, task
@task(persist_result=True)
def compute():
return 42
@flow(persist_result=True)
def pipeline():
return compute()
Prod (GCS-backed results)
- Goal: results land in GCS; no host coupling; works across nodes.
- Image/worker prerequisites:
- Install the GCS extras:
pip install "prefect[gcp]"
- Provide GCP credentials (ADC). Either:
- Workload Identity (best in GKE), or
- Mount a service account JSON and set GOOGLE_APPLICATION_CREDENTIALS
to its path.
- Configure default result storage to a GCS URL and enable persistence:
await flow.deploy(
name="huscarl-prod",
work_pool_name="my-podman-pool",
image="your-prod-image", # same code-baked image is fine
build=False,
push=False,
job_variables={
"env": {
"PREFECT_RESULTS_PERSIST_BY_DEFAULT": "true",
"PREFECT_RESULTS_DEFAULT_STORAGE": "<gcs://my-prefect-results/huscarl/>",
# if using a mounted SA key:
# "GOOGLE_APPLICATION_CREDENTIALS": "/var/secrets/gcp/sa.json",
},
# If using a mounted SA key file:
# "volumes": ["/secure/path/sa.json:/var/secrets/gcp/sa.json:ro,Z"],
},
)
- The UI will show GCS URLs for persisted results; you can fetch them with gcloud/gsutil or programmatically.
Why this works
- Prefect 3 persists results when enabled and writes them to a “result storage.” If you don’t set one, it defaults to a local path inside the container (e.g., /root/.prefect/storage/...), which other machines can’t see. Setting PREFECT_RESULTS_DEFAULT_STORAGE
to a real path (local mount in dev, GCS URL in prod) makes the paths valid where the code runs.
A couple of sanity checks
- Ensure your flow/tasks actually request persistence (persist_result=True
) or set PREFECT_RESULTS_PERSIST_BY_DEFAULT=true
as shown.
- For Podman on SELinux, keep :Z
on volume mounts.
- Your custom image should include any libs needed to serialize your results.
Docs, for when you’re feeling adventurous
- Prefect 3: Results (persistence and storage)
If you share your bucket name and how you supply GCP creds (WI vs key file), I’ll give you the exact job_variables
snippet for your environment.Paulien Jeunesse
08/15/2025, 5:27 PMMarvin
08/15/2025, 5:27 PMMarvin
08/15/2025, 5:29 PMmkdir -p /srv/prefect-home
export PREFECT_HOME=/srv/prefect-home
prefect server start
(If your server is already running, restart it with PREFECT_HOME set.)
2) Mount the same host path into your flow-run container at the same absolute path, and set PREFECT_HOME in the container
await flow.deploy(
name="huscarl-dev",
work_pool_name="my-podman-pool",
image="your-custom-image",
build=False,
push=False,
job_variables={
"volumes": ["/srv/prefect-home:/srv/prefect-home:Z"], # :Z for Podman/SELinux
"env": {
"PREFECT_HOME": "/srv/prefect-home",
"PREFECT_RESULTS_PERSIST_BY_DEFAULT": "true",
},
},
)
Why this works
- Prefect writes local results under ${PREFECT_HOME}/storage/...
.
- If the server and the flow-run container don’t share that path, the server can’t read the files and you get the “does not exist” error.
- By using the same absolute path on both sides, the files are visible to both.
Alternative you might actually prefer
- Use GCS for dev as well. Then there’s no shared filesystem to keep in sync. Just set:
- In your deployment `job_variables.env`:
- PREFECT_RESULTS_PERSIST_BY_DEFAULT=true
- PREFECT_RESULTS_DEFAULT_STORAGE=<gcs://your-bucket/dev/>
- plus GCP creds (ADC via Workload Identity or GOOGLE_APPLICATION_CREDENTIALS
to a mounted key)
- The server will fetch results from GCS directly.
Docs for reference: Prefect 3: Results
If you share where your server is running (host path to its PREFECT_HOME) I’ll give you the exact volume/env snippet to match it.Paulien Jeunesse
08/15/2025, 6:00 PMMarvin
08/15/2025, 6:00 PMMarvin
08/15/2025, 6:01 PMmodel_dump()
, or your own .to_dict()
).
Example:
from prefect import flow, task
@task(persist_result=True)
def compute():
# Convert custom types to plain JSON-friendly data first
return {"status": "ok", "value": 42}
@flow(persist_result=True)
def fake_workflow():
return compute()
- Result previews in the UI will work without your package installed on the server.
B) Make the server able to unpickle your custom objects
- Install the same package/code in the environment that runs prefect server start
.
- If you run the server on your host:
pip install -e /path/to/your/repo # where 'prefect_fake' lives
- If you run the server in a container, bake your package into that image or mount it and set PYTHONPATH
accordingly.
- This is workable for dev, but brittle for prod. Prefer A.
Notes
- Using GCS for prod doesn’t change this behavior: the server still needs to deserialize to preview. So either store JSON-friendly values or install your package in the server environment.
- If you don’t need previews, you can persist raw artifacts separately or skip result persistence for those tasks.
Quick questions to tailor the exact steps:
- Is prefect_fake
your package/module name?
- How are you running the Prefect server (host venv vs container)?
- What do your tasks return that needs pickling? If you share a snippet, I’ll show a minimal JSON-friendly conversion.Paulien Jeunesse
08/15/2025, 6:13 PMMarvin
08/15/2025, 6:13 PMMarvin
08/15/2025, 6:14 PMpodman build -t your/image:tag .
(as the same user that runs the worker) or podman pull your/image:tag
- Use the same user/engine for build and run (avoid root/rootless mismatch). If that’s not possible, push to a registry and pull.
- Keep images lean; install everything at build time, not on container start.
- Ensure the image tag in your deployment matches exactly what’s on the worker host.
- Start more worker concurrency if you’re queueing.
- Turn up worker logs to see where time goes:
PREFECT_LOGGING_LEVEL=DEBUG prefect worker start --pool my-podman-pool
You’ll see lines like “Pulling image…” or “Creating container…”, which correlate directly with the long Pending.
If you tell me:
- the exact image tag in your deployment,
- whether the worker is rootless Podman or rootful,
- and whether Pending is slow only on first run or every run,
I’ll point at the precise bottleneck (usually “image not present for that user” or a slow pull).Paulien Jeunesse
08/15/2025, 6:17 PMMarvin
08/15/2025, 6:17 PMMarvin
08/15/2025, 6:18 PM# start your worker with a shorter poll interval (seconds)
PREFECT_WORKER_POLLING_INTERVAL=2 PREFECT_LOGGING_LEVEL=DEBUG prefect worker start --pool my-podman-pool
- Or set it in your Prefect profile/env permanently (same env var).
- Keep it reasonable (2–5s). Lower = more API chatter.
And remember: even after claim, Pending still covers image pulls and container creation. Pre-pull your image on the worker host (as the same user the worker runs under) to avoid that:
podman pull your/image:tag
# or build locally with that user so it’s already present
podman build -t your/image:tag .
If you want, share how you’re launching the worker (command/env) and I’ll give you the exact incantation for your setup.