Tinh
12/17/2025, 2:58 PMdeploy a flow in Prefect 3.2.6
There are in total 20 tasks, running in 7 flows . Among the task, there are 8 tasks running inside container, using locally built docker image.
Could you pls guide me how to get my flow run in
flow.serve()Marvin
12/17/2025, 2:59 PMMarvin
12/17/2025, 3:04 PMfrom prefect import flow
@flow(log_prints=True)
def my_flow(name: str = "world"):
print(f"Hello, {name}!")
if __name__ == "__main__":
my_flow.serve(
name="hello-deployment",
cron="0 9 * * *", # every day at 9 AM
parameters={"name": "Alice"},
tags=["production"],
pause_on_shutdown=False # keep schedule active if this process restarts
)
Serve multiple flows together
import time
from prefect import flow, serve
@flow
def slow_flow(sleep: int = 30):
time.sleep(sleep)
@flow
def fast_flow():
return "done"
if __name__ == "__main__":
slow_dep = slow_flow.to_deployment(
name="sleeper",
interval=45
)
fast_dep = fast_flow.to_deployment(
name="fast",
cron="*/5 * * * *"
)
# Run both deployments in one long-lived process
serve(slow_dep, fast_dep, limit=10) # limit = max concurrent runs across these deployments
Common options you can pass to serve()/to_deployment()
- name: deployment name (no slashes)
- interval / cron / rrule / schedules: schedule configuration
- parameters: default run parameters
- tags, description, version
- global_limit: limit concurrent runs for a deployment
- limit (on serve): total concurrent runs across all served deployments in this process
- pause_on_shutdown: pause schedules when the process stops (default True)
Running inside your own Docker container (if you want a long-lived “runner” container)
- This is valid with flow.serve(): you build an image with your code, run it, and it serves your deployments locally within that container.
- Example Dockerfile:
# Dockerfile
FROM python:3.11-slim
WORKDIR /app
RUN pip install --no-cache-dir "prefect==3.2.6" # plus your deps
COPY . /app
# If your tasks talk to Docker, you’ll likely mount the host docker.sock at runtime:
# docker run -v /var/run/docker.sock:/var/run/docker.sock ...
CMD ["python", "app.py"] # app.py calls flow.serve(...)
- If your tasks need Docker access from inside this container, run it with:
- Linux: mount /var/run/docker.sock and grant permissions
- Windows/Mac: ensure Docker Desktop exposes the daemon to the container
Important differences: serve() vs deploy()
- serve()
- Runs everything locally (or inside your long-lived container)
- No work pools, no Docker image per run
- Great for local dev or a simple self-hosted runner
- deploy()
- Registers deployments that run on dynamic infrastructure via work pools (Docker, K8s, etc.)
- Supports specifying image, job variables, scaling, etc.
- Use this if you must run each flow in a specific Docker image per run
Connectivity checklist
- If you are using Prefect Cloud or a remote Prefect Server, set:
- PREFECT_API_URL and (for Cloud) PREFECT_API_KEY in the environment where flow.serve() runs
- For local testing with Prefect Server:
- Start it with: prefect server start
- By default, PREFECT_API_URL is set automatically for local runs
- You’ll see the deployment appear in the UI once serve() starts
Next stepsMarvin
12/17/2025, 3:04 PMTinh
12/17/2025, 3:38 PMDocker container has no permission to access the file system (mounted by -v local-path:/docker-path
how could I do then?
my workflow run on local VM and will use local imageMarvin
12/17/2025, 3:38 PMMarvin
12/17/2025, 3:39 PMchmod +x as needed.
- Align ownership/permissions:
sudo chown -R 1000:1000 /host/path
sudo chmod -R u+rwX /host/path
…or run the container as the matching UID/GID:
docker run --user $(id -u):$(id -g) ...
- SELinux (Fedora/RHEL/CentOS)
- Use the :z (shared) or :Z (private) label on the volume:
- CLI: -v /host/path:/container/path:rw,z
- Docker SDK: volumes={'/host/path': {'bind': '/container/path', 'mode': 'rw,Z'}}
- Docker Desktop (Mac/Windows)
- Make sure the directory/drive is shared in Docker Desktop settings.
- Running Prefect inside a container and launching more containers (Docker-outside-of-Docker)
- The bind source is resolved by the host Docker daemon, not by the inner container. You must pass a host path that exists on the VM host, not a path inside your first container.
- Solutions:
- Mount the host directory into the “runner” container at the exact same absolute path as on the host so you can pass the same path to child containers, or
- Use a named Docker volume, or
- Run Prefect directly on the VM (no nested containers).
Sanity-check commands
- Plain test of your mount and permissions:
docker run --rm -it \
-v /host/path:/work:rw,z \
--user $(id -u):$(id -g) \
alpine:3.20 \
sh -lc "whoami && id && ls -la /work && touch /work/testfile"
If this fails, it’s a host perms/SELinux/file-sharing issue.
- If you run Prefect inside a long-lived container and spawn other containers, run the runner like this:
docker run -d --name prefect-runner \
-e PREFECT_API_URL=... \
-e PREFECT_API_KEY=... \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /data/shared:/data/shared:rw,z \
# IMPORTANT: keep the same absolute path inside and outside
-v /abs/host/path:/abs/host/path:rw,z \
my-runner-image:latest
Then, when launching child containers, use /abs/host/path as the bind source (not /data/shared), because the host daemon resolves it.
Example Prefect task using Docker SDK with correct volume options
import os
import docker
from prefect import task
@task
def run_in_container(host_path: str):
client = docker.from_env()
container = client.containers.run(
"my-local-image:latest",
command=["sh", "-lc", "ls -la /work && touch /work/test.txt"],
volumes={
host_path: {"bind": "/work", "mode": "rw,Z"} # add Z if SELinux; otherwise "rw"
},
user=f"{os.getuid()}:{os.getgid()}", # align UID/GID with host ownership
remove=True,
tty=True,
)
If you’re still blocked, please share:
- The exact docker run ... (or docker SDK) command you’re using (with the volume flags)
- Whether you’re on Linux vs Docker Desktop
- Whether you’re running Prefect directly on the VM or inside a container
- A snippet of the error message (Permission denied path, etc.)
Helpful references
- Docker bind mounts
- SELinux labels on mounts
- Docker Desktop file sharing (Mac)
- Docker Desktop file sharing (Windows)
- Prefect: run flows in local processes (flow.serve)