https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I am working on setting up a self hos...
# ask-marvin
o
@Marvin I am working on setting up a self hosted Prefect setup from my Windows machine and currently have a server, worker pool and worker running on Docker. My issue is that the pipelines I want to convert into flows work with a .duckdb database file. As that is something that will be constantly changing with new inserts and so on (mostly from prefect orchestrated jobs) I don't think it makes sense to include this database file in a static Docker image? How can I proceed? Is it possible to use something like mounting the database file with the flow runs?
m
✅ thought for 316.2 seconds
Yes—don’t bake the DuckDB file into your image. With a Docker work pool/worker in Prefect 3.x, you can bind-mount a host directory (or file) into each flow-run container via job variables. That way your runs see and update the same persistent DuckDB file on the host. Here are the common approaches (pick one): 1) Bind-mount a host directory that contains the DuckDB file (recommended) - Windows path tip: use forward slashes or double backslashes and quote the string. - Prefer mounting the containing directory rather than a single file to avoid path quirks. Example deployment (per-deployment job variables):
Copy code
from prefect import flow

@flow
def my_flow():
    # Inside the container, refer to the mounted path
    db_path = "/data/my.duckdb"
    # use duckdb.connect(db_path) or your connector of choice

my_flow.from_source(
    source=".",                  # or a repo URL
    entrypoint="flows/my_flow.py:my_flow",
).deploy(
    name="duckdb-etl",
    work_pool_name="docker-pool",
    image="your-repo/your-image:latest",
    job_variables={
        "volumes": [
            "C:/prefect-data:/data"  # Mount host folder C:\prefect-data into /data
            # optionally add :ro or :rw, default is rw
        ],
        # optional: "env": {"DUCKDB_PATH": "/data/my.duckdb"}
    },
)
You can also set this once at the pool level so all deployments inherit it: - Prefect UI -> Work Pools -> your Docker pool -> Edit Base Job Template -> add a 
volumes
list like:
Copy code
{
  "volumes": [
    "C:/prefect-data:/data"
  ]
}
Or inspect the available fields with:
Copy code
prefect work-pool get-default-base-job-template --type docker
You’ll see 
volumes
is a supported field. 2) Use a Docker named volume - Define a named volume (e.g., 
prefect_duckdb
) and mount it for all flow runs:
Copy code
job_variables = {
  "volumes": ["prefect_duckdb:/data"]
}
- This keeps data inside Docker’s VM (with Docker Desktop on Windows). It’s great if all read/write access happens from containers. It’s less convenient if you need to open the file directly on the Windows host. Important Windows notes - Ensure your C: drive is shared in Docker Desktop. - Use paths like 
C:/path/to/dir:/data
or escape backslashes 
C:\\path\\to\\dir:/data
. - Prefer mounting a directory, then refer to the file inside it (e.g., 
/data/my.duckdb
). - If your worker itself runs in Docker, it must have access to the host Docker daemon (the standard Prefect Docker worker container mounts the Docker socket). The bind mount path is resolved on the Docker host, not inside the worker container. Concurrency and DuckDB safety - DuckDB supports many readers but typically only one writer at a time per file. If multiple flows write to the same file, serialize writes. - In Prefect, you can enforce this with a concurrency limit on a tag for writer tasks:
Copy code
prefect concurrency-limit create duckdb-writer --limit 1
Then tag your write operations with 
duckdb-writer
. Docs - Customize job variables: Customize job variables (Prefect 3.x) If you share a bit more about where the DuckDB file lives today (host path, shared drive, etc.) and whether all writers will run as Prefect-managed Docker containers on the same machine, I can suggest the exact volume string and the cleanest setup.
o
@Marvin I will be needing to read often also from outside of the prefect and docker, do you suggest option 1 then? Any glaring issues with that, or should that alternative both support reading/writing inside of Docekr Desktop and on my regular computer?
m
✅ thought for 48.9 seconds
Yes—go with option 1. A bind-mounted host directory is the best fit when you need to open the DuckDB file from both: - Prefect-managed Docker flow runs, and - your regular Windows apps outside Docker That gives you a single “source of truth” DuckDB file on your Windows filesystem, and each flow-run container sees it at a stable mount point. How to set it up - Pick a stable host directory without spaces (e.g., 
C:\prefect-data\duckdb
) - Mount that directory into your flow-run containers (e.g., 
/data
) - Inside your flows, always reference the DB via the container path (e.g., 
/data/my.duckdb
) - Outside Docker, use the Windows path (e.g., 
C:\prefect-data\duckdb\my.duckdb
) Example deployment using a Docker work pool
Copy code
my_flow.from_source(
    source=".",  # or your repo URL
    entrypoint="flows/my_flow.py:my_flow",
).deploy(
    name="duckdb-etl",
    work_pool_name="docker-pool",
    image="your-repo/your-image:latest",
    job_variables={
        "volumes": [
            "C:/prefect-data/duckdb:/data"
        ],
        # optional: "env": {"DUCKDB_PATH": "/data/my.duckdb"}
    },
)
Or set it once at the pool level so all deployments inherit it: - UI -> Work Pools -> your Docker pool -> Edit Base Job Template -> add
Copy code
{
  "volumes": [
    "C:/prefect-data/duckdb:/data"
  ]
}
Notes and caveats on Windows - Use 
C:/path/...:/container/path
or escape backslashes 
C:\\path\\...:/container/path
- Ensure your C: drive is shared in Docker Desktop - Prefer mounting the directory that contains the 
.duckdb
file, not the file itself - Avoid spaces in the host path to sidestep “invalid volume specification” issues Concurrency and safety with DuckDB - DuckDB supports many readers but typically only one writer. If multiple flows (or your host apps) might write concurrently, serialize writes. - In Prefect, you can enforce this with a concurrency limit tag: - 
prefect concurrency-limit create duckdb-writer --limit 1
- Tag writer tasks with 
duckdb-writer
- If you also write from outside Prefect, try to avoid overlapping with container writes, or implement retry logic on “database is locked” errors. Why not a Docker named volume? - Named volumes live inside Docker Desktop’s VM, which makes host access awkward. Since you need regular host access, bind-mounting a Windows directory is the right choice. Docs - Customize job variables (including volumes): Customize job variables (Prefect 3.x) If you share the exact host path you plan to use and your pool name, I can give you a ready-to-paste 
job_variables
block and confirm the flow code path references.
o
@Marvin pydantic_core._pydantic_core.ValidationError: 1 validation error for DockerWorkerJobConfiguration volumes.0 Value error, Invalid volume option: '/data'. Must be 'ro' or 'rw' [type=value_error, input_value='C/qfin/qfin/database/data', input_type=str] For further information visit https://errors.pydantic.dev/2.11/v/value_error i tried giving _job_variables_={ 'volumes': [ 'C/qfin/qfin/database/data', ], 'env': { 'QFIN_DATABASE_DUCK': '/data/database.duckdb', }, },
m
✅ thought for 152.9 seconds
You’re running into a classic Windows-path + Linux Docker client parsing issue. What’s happening - Your Docker worker is running in a Linux container and uses the Python Docker SDK, which parses volume strings by splitting on “:”. - A Windows path like 
C:/...:/data
becomes three parts (
C
, 
/...
, 
/data
). The parser thinks 
/data
is the “mode” and errors because mode must be 
ro
or 
rw
. Two solid fixes (pick one) A) Use the Docker Desktop Linux host path form - On Windows with Docker Desktop (Linux engine), host drives are exposed under 
/host_mnt/<drive-letter>/...
. - Use that Linux-style path and include an explicit mode, e.g. 
:rw
. Example:
Copy code
job_variables = {
    "volumes": [
        "/host_mnt/c/qfin/qfin/database:/data:rw"
    ],
    "env": {
        "QFIN_DATABASE_DUCK": "/data/database.duckdb"
    },
}
Notes: - Ensure your C: drive is shared in Docker Desktop. - Inside your flows, use 
/data/database.duckdb
. - Outside Docker, use 
C:\qfin\qfin\database\database.duckdb
. B) Mount the parent directory instead of a single file - This avoids edge cases and gives you flexibility if you add more files later. - Same syntax as above, just point to the directory (as you already do). Why not keep the Windows 
C:/...
format? - The worker’s Linux-side parser will always misread the 
C:
colon as a separator. - Using 
/host_mnt/c/...
sidesteps that and aligns with the Linux Docker engine’s view of your Windows filesystem. Optional: set once at the pool level - Prefect UI -> Work Pools -> your Docker pool -> Edit Base Job Template -> add:
Copy code
{
  "volumes": [
    "/host_mnt/c/qfin/qfin/database:/data:rw"
  ]
}
Docs - Customize job variables: Customize job variables (Prefect 3.x) If you try the 
/host_mnt/...
form and still see issues, let me know your exact pool name and the full 
job_variables
you’re using; I’ll tailor a ready-to-paste block.
o
@Marvin got a little further but still having issues: Unexpected exception encountered when trying to load flow Traceback (most recent call last): File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/deployments/steps/core.py", line 166, in run_steps step_output = await run_step(step, upstream_outputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/deployments/steps/core.py", line 137, in run_step result = await from_async.call_soon_in_new_thread( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 374, in aresult return await asyncio.wrap_future(self.future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 399, in _run_sync result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/deployments/steps/pull.py", line 35, in set_working_directory os.chdir(directory) FileNotFoundError: [Errno 2] No such file or directory: 'C:\\qfin\\qfin' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/engine.py", line 112, in <module> flow: "Flow[..., Any]" = load_flow(flow_run) ^^^^^^^^^^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/flow_engine.py", line 150, in load_flow flow = run_coro_as_sync( ^^^^^^^^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync return call.result() ^^^^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result return self.future.result(timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async result = await coro ^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper return await task ^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/client/utilities.py", line 69, in wrapper return await func(client, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/flows.py", line 2810, in load_flow_from_flow_run output = await run_steps( ^^^^^^^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/deployments/steps/core.py", line 194, in run_steps raise StepExecutionError(f"Encountered error while running {fqn}") from exc prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.set_working_directory I am using WSL2 in Docker Desktop, so folders should be shared by default
m
✅ thought for 281.1 seconds
Thanks for the traceback — that points to a different problem than the volume mount. The failure is coming from the deployment’s “set_working_directory” step trying to chdir to a Windows path inside a Linux container: - set_working_directory → os.chdir("C:\\qfin\\qfin") → FileNotFoundError Why this happens - When you created the deployment, Prefect captured your local source directory (a Windows path). - At run time, the Docker worker spins up a Linux container. That Windows path does not exist in the container, so the “pull” step fails before your flow code is even loaded. This is unrelated to WSL2/drive sharing—the issue is the Windows path recorded in the deployment’s source metadata. How to fix it (pick one) Option A: Use a Git repo as the source (recommended when using Docker workers) - Put your flow code in a Git repo (public or private). - Deploy with a Git URL so the worker can clone the code into a container-native path at runtime:
Copy code
my_flow.from_source(
    source="<https://github.com/OWNER/REPO>",    # or your git server URL
    entrypoint="path/to/flow_file.py:my_flow",
).deploy(
    name="duckdb-etl",
    work_pool_name="docker-pool",
    image="your-repo/your-image:latest",
    job_variables={
        "volumes": ["/host_mnt/c/qfin/qfin/database:/data:rw"],
        "env": {"QFIN_DATABASE_DUCK": "/data/database.duckdb"},
    },
)
- If the repo is private, add git credentials via 
job_variables["env"]
or your image. Option B: Bundle your code into the image (no Windows path recorded) - Build an image that copies your project into the image and sets a Linux workdir (e.g., 
/app
), then use that image in your deployment. Example Dockerfile:
Copy code
FROM python:3.12-slim
WORKDIR /app
COPY . /app
RUN pip install -U pip && pip install -U prefect duckdb <other deps>
- Build and push the image, then deploy referencing that image (and do not point Prefect at a local Windows path for 
source
):
Copy code
my_flow.deploy(
    name="duckdb-etl",
    work_pool_name="docker-pool",
    image="your-repo/your-image:latest",
    job_variables={
        "volumes": ["/host_mnt/c/qfin/qfin/database:/data:rw"],
        "env": {"QFIN_DATABASE_DUCK": "/data/database.duckdb"},
        # optional: "container_create_kwargs": {"working_dir": "/app"}
    },
)
- This avoids the “pull from local Windows path” step entirely because the code is already in the image. Volume reminder for Windows + Docker Desktop - Keep using the Linux-style path and explicit mode: - 
"/host_mnt/c/qfin/qfin/database:/data:rw"
- Inside your flow, read/write 
/data/database.duckdb
. - On your host, use 
C:\qfin\qfin\database\database.duckdb
. After you switch to A or B, re-create the deployment so the old Windows path isn’t embedded in its steps. Reference - Customize job variables (e.g., volumes): Customize job variables (Prefect 3.x) If you share your current deployment code (the exact 
from_source(...).deploy(...)
snippet), I can point out the minimal change needed and provide a ready-to-paste version.
o
@Marvin issue with .flow_from_source() with github source is i run into issues on windows platform: my_flow = flow.from_source( ^^^^^^^^^^^^^^^^^ File "C:\qfin\qfin\.venv\Lib\site-packages\prefect\_internal\compatibility\async_dispatch.py", line 94, in wrapper return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "C:\qfin\qfin\.venv\Lib\site-packages\prefect\flows.py", line 1365, in from_source run_coro_as_sync(storage.pull_code()) File "C:\qfin\qfin\.venv\Lib\site-packages\prefect\utilities\asyncutils.py", line 207, in run_coro_as_sync return call.result() ^^^^^^^^^^^^^ File "C:\qfin\qfin\.venv\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 365, in result return self.future.result(timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\qfin\qfin\.venv\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 192, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "C:\python\anaconda312\Lib\concurrent\futures\_base.py", line 401, in __get_result raise self._exception File "C:\qfin\qfin\.venv\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 441, in _run_async result = await coro ^^^^^^^^^^ File "C:\qfin\qfin\.venv\Lib\site-packages\prefect\utilities\asyncutils.py", line 188, in coroutine_wrapper return await task ^^^^^^^^^^ File "C:\qfin\qfin\.venv\Lib\site-packages\prefect\runner\storage.py", line 348, in pull_code await self._clone_repo() File "C:\qfin\qfin\.venv\Lib\site-packages\prefect\runner\storage.py", line 384, in _clone_repo await run_process(cmd) File "C:\qfin\qfin\.venv\Lib\site-packages\anyio\_core\_subprocesses.py", line 91, in run_process async with await open_process( ^^^^^^^^^^^^^^^^^^^ File "C:\qfin\qfin\.venv\Lib\site-packages\anyio\_core\_subprocesses.py", line 190, in open_process return await get_async_backend().open_process( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\qfin\qfin\.venv\Lib\site-packages\anyio\_backends\_asyncio.py", line 2567, in open_process process = await asyncio.create_subprocess_exec( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\python\anaconda312\Lib\asyncio\subprocess.py", line 224, in create_subprocess_exec transport, protocol = await loop.subprocess_exec( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\python\anaconda312\Lib\asyncio\base_events.py", line 1743, in subprocess_exec transport = await self._make_subprocess_transport( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\python\anaconda312\Lib\asyncio\base_events.py", line 524, in _make_subprocess_transport raise NotImplementedError NotImplementedError I use UV as package manager, so might be related to that. not sure
m
✅ thought for 68.2 seconds
Great detail — thank you. That traceback is a Windows asyncio/subprocess issue, not a Prefect bug. Cause - 
flow.from_source(... )
needs to run 
git clone
(via an async subprocess) to pull your repo locally. - On Windows, the default asyncio event loop policy may not support subprocesses, leading to 
NotImplementedError
when 
create_subprocess_exec
is called. Quickest fix on Windows - Set the asyncio loop policy to one that supports subprocesses before you call 
from_source
. Add this at the very top of the script where you’re calling `from_source`:
Copy code
import sys
import asyncio

if sys.platform == "win32":
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
Then your deployment code can run as-is:
Copy code
from prefect import flow

@flow
def my_flow():
    ...

my_flow.from_source(
    source="<https://github.com/OWNER/REPO>",
    entrypoint="path/to/flow.py:my_flow",
).deploy(
    name="duckdb-etl",
    work_pool_name="docker-pool",
    image="your-repo/your-image:latest",
    job_variables={
        "volumes": ["/host_mnt/c/qfin/qfin/database:/data:rw"],
        "env": {"QFIN_DATABASE_DUCK": "/data/database.duckdb"},
    },
)
Notes - Make sure 
git
is installed and on your PATH in the Windows environment running the deploy. - Using uv as a package manager should not affect this; the event loop policy is the key here. - Alternatively, run the deploy command from WSL/Linux (where subprocess support is native), or use the “code in image” pattern (bundle your flow code into the image and call 
flow.deploy(...)
without 
from_source
). You’re already on the right track with the volume: - Inside containers: use 
/data/database.duckdb
- On Windows host: 
C:\qfin\qfin\database\database.duckdb
- Keep using 
"/host_mnt/c/qfin/qfin/database:/data:rw"
for the mount. If you still see issues after setting the loop policy, share your deploy script snippet (top ~15 lines and the 
from_source(...).deploy(...)
call) and I’ll tailor a ready-to-paste version.
o
@Marvin I deployed using image with flow included. When running, I get: Unexpected exception encountered when trying to load flow Traceback (most recent call last): File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/engine.py", line 112, in <module> flow: "Flow[..., Any]" = load_flow(flow_run) ^^^^^^^^^^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/flow_engine.py", line 150, in load_flow flow = run_coro_as_sync( ^^^^^^^^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync return call.result() ^^^^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 365, in result return self.future.result(timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 441, in _run_async result = await coro ^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper return await task ^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/client/utilities.py", line 69, in wrapper return await func(client, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/flows.py", line 2800, in load_flow_from_flow_run await storage_block.get_directory(from_path=from_path, local_path=".") File "/opt/prefect/app/.venv/lib/python3.12/site-packages/prefect/filesystems.py", line 162, in aget_directory copytree(from_path, local_path, dirs_exist_ok=True, ignore=ignore_func) File "/usr/local/lib/python3.12/shutil.py", line 598, in copytree with os.scandir(src) as itr: ^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/app/None'
m
✅ thought for 88.6 seconds
Thanks — this error means the deployment is still trying to “pull code from storage” at run time, even though your code is already baked into the image. It’s attempting to copy from a storage path that resolves to /opt/prefect/app/None, hence the FileNotFoundError. Why this happens - If a deployment was created with from_source (or otherwise configured with storage/pull steps), those steps get embedded into the deployment. Switching to “code in image” without recreating the deployment can leave those stale pull steps in place. What to do 1) Recreate the deployment without from_source and without storage - Do not call from_source when you’re using “code in image”. - Import your flow normally and call deploy on the flow object. - Use a new deployment name or delete the old one first so you don’t inherit stale steps. Minimal working pattern - Dockerfile
Copy code
FROM python:3.12-slim
WORKDIR /app
COPY . /app
RUN pip install -U pip && pip install -U prefect duckdb  # + your deps
- Deploy script (run on your Windows host)
Copy code
from prefect import flow
from flows.my_flow import my_flow  # adjust to your project structure

# Important: do NOT use from_source here
my_flow.deploy(
    name="duckdb-etl",                      # new name to avoid stale config
    work_pool_name="docker-pool",
    image="your-repo/your-image:latest",    # already built/pushed image
    build=False,
    push=False,
    job_variables={
        "container_create_kwargs": {"working_dir": "/app"},
        "volumes": ["/host_mnt/c/qfin/qfin/database:/data:rw"],
        "env": {"QFIN_DATABASE_DUCK": "/data/database.duckdb"},
    },
)
Key checks - In the Prefect UI, open the deployment and inspect the “Pull steps”. For a “code in image” deployment, you should NOT see a “pull from storage” step. You should see a “load from entrypoint” step. If you still see storage steps, delete the deployment and recreate it with the code above. - Ensure that the flow file path inside the image matches the entrypoint Prefect records. With the example above, Prefect will record something like flows/my_flow.py:my_flow. That file must exist at /app/flows/my_flow.py in the image (we set working_dir to /app). Windows + volume reminder - Keep using Linux-style path + explicit mode: - "/host_mnt/c/qfin/qfin/database/datarw" - Inside container: use /data/database.duckdb - On host: C:\qfin\qfin\database\database.duckdb Optional - Upgrade Prefect to the latest 3.x on your dev machine and in the image to avoid older storage/pull-step quirks: - uv add -U prefect Docs - Customize job variables (e.g., volumes, container_create_kwargs): Customize job variables (Prefect 3.x) If you can share: - Your current Dockerfile - The exact deploy snippet you used I can confirm the entrypoint and provide a ready-to-paste corrected deployment.
30 Views
Open in Slack
PreviousNext
Powered by