✅ thought for 503.0 seconds

Thanks for the details — a few likely causes come to mind. Quick context: In Prefect 3.x,

.serve()

registers your flow as a “runner deployment” and executes runs in the same Python process where you call

.serve()

. It does not use a work pool or spin up Docker/K8s infrastructure for flow runs. That means any task that launches containers is relying on Docker being reachable from that exact process and environment. See: Schedule your code quickly with flow.serve() and How to: Serve flows in Docker. Common reasons container tasks fail under `.serve()`: - Docker is not reachable from the process running

.serve()

- Docker Desktop not running (Mac/Windows), or Docker Engine not installed (Linux) - Missing permissions to access the Docker socket (

/var/run/docker.sock

) — common on Linux unless the user is in the

docker

group - Running

.serve()

inside a container without mounting the Docker socket into that container - Missing

prefect[docker]

extra in the environment running

.serve()

- Image availability - The tag you use exists only on your dev machine; the environment running

.serve()

cannot find/pull it - Fix: push to a registry and pull, or pre-pull on that machine - Volume mounts or paths - Relative paths work when you run your script, but under

.serve()

the working directory differs; container mounts then fail. Use absolute paths. - “localhost” networking assumptions - If a container needs to reach your Prefect API (or other services) and you’re pointing at

<http://localhost>:...

, from inside the container that resolves to the container itself, not your host - Use

host.docker.internal

on Mac/Windows or add a host-gateway on Linux (

--add-host=host.docker.internal:host-gateway

) and set

PREFECT_API_URL

accordingly Two ways to move forward: A) Keep using

.serve()

and make Docker reachable in that environment - Ensure Docker is installed and running, and the

.serve()

user can access it - If

.serve()

runs in a container, mount the Docker socket and grant permissions: -

-v /var/run/docker.sock:/var/run/docker.sock

- Install the Prefect Docker extra where

.serve()

runs: -

pip install "prefect[docker]"

- Use absolute paths for any volume mounts - If your container needs to talk to the Prefect API, set

PREFECT_API_URL

inside the container to a reachable host (avoid plain

localhost

) B) If you want each flow run to execute in a container, use a Docker work pool instead of

.serve()

- Create a Docker work pool:

prefect work-pool create "docker-pool" --type docker

- Deploy to the pool with an image:

from prefect.flows import Flow

  # Use a git repo or local project as the source of your flow code
  Flow.from_source(".", "path/to/deploy.py:your_flow").deploy(
      name="my-docker-deploy",
      work_pool_name="docker-pool",
      image="your-registry/your-image:tag",
      build=True,
      push=True,
  )

- Start a Docker worker (ensure docker extras are available):

prefect worker start -p docker-pool

If you prefer ensuring extras at runtime:

uv run --with 'prefect[docker]' prefect worker start -p docker-pool

This pattern runs each flow run inside a container managed by the Docker worker rather than your local Python

.serve()

process. Can you share: - One failing traceback from a “task with container” - How you launch containers (prefect-docker tasks, docker SDK, shell commands, etc.) - Where you’re running

.serve()

(host OS vs. inside a container), and whether Docker is installed/reachable there - Any volume mounts and whether they’re absolute or relative - Whether those images are only local or pushed to a registry With that, I can point you to the exact fix.

Thanks for the details! Short answer: with

.serve()

your flow runs execute in the same local Python process that’s running

serve

(no worker infrastructure). That means any subflows you call directly will inherit that same local environment. They won’t automatically run inside your Docker images — which is likely why your “container” steps fail when switching to

.serve()

. A few key points and ways to fix it: Why this happens - In Prefect 3.x, infrastructure is applied per flow run, not per task. If you call a subflow like a normal function, it runs in the same environment as the parent. Under

.serve()

, that’s your local process, not Docker. - If your “container tasks” rely on Docker-only dependencies (or your code expects to be in a container), running under

.serve()

will fail unless you explicitly start containers from your tasks (via Docker SDK/CLI) and Docker is available to that process. Two solid patterns to run parts of your workflow in Docker 1) Split into deployments and trigger the Docker subflow via a deployment - Put the containerized part into its own flow and bind it to a Docker work pool so it runs in your image. - From the parent flow, call it via

run_deployment

so it executes on the Docker worker (i.e., in the container), not in the local

.serve()

process. - Docs: Serve flows with Docker Example setup: - Create a Docker work pool and start a Docker worker (with docker extras installed):

uv run --with 'prefect[docker]' prefect work-pool create -t docker docker-pool
     uv run --with 'prefect[docker]' prefect worker start -p docker-pool

- Serve the Dockerized subflow bound to that pool:

from prefect import flow
     from prefect.flows import serve, bind_flow_to_infrastructure
     from prefect_docker.worker import DockerWorker

     @flow
     def docker_subflow(x: int) -> int:
         # code that requires your Docker image environment
         return x + 1

     if __name__ == "__main__":
         serve(
             bind_flow_to_infrastructure(
                 docker_subflow,
                 work_pool="docker-pool",
                 worker_cls=DockerWorker,
                 job_variables={
                     "image": "your-repo/your-image:tag",
                     # optional:
                     # "env": {"MY_VAR": "value"},
                     # "volumes": ["/host/path:/container/path"],
                     # "networks": ["my_docker_network"],
                 },
             )
         )

- In the parent flow (served locally), trigger the subflow as a deployment:

from prefect import flow
     from prefect.deployments import run_deployment
     from prefect.flow_runs import wait_for_flow_run

     @flow
     def parent_flow():
    #   ... do local work ...
         fr = run_deployment("docker_subflow/docker_subflow", parameters={"x": 10})
         wait_for_flow_run(fr)  # waits for the containerized subflow to finish

This way, your parent can still use

.serve()

locally, while the Docker subflow reliably runs in a container via the Docker worker. 2) If you must keep subflows as direct calls, run Docker from inside tasks - Keep everything in one flow served locally, but have the “container tasks” create and run containers explicitly using the Docker SDK/CLI (e.g., via

prefect-docker

collection). - Make sure: - Docker is installed on the machine running

.serve()

and the

prefect

process has permission to access the Docker daemon. - Image names are available locally or can be pulled. - Volumes, networks, and env vars are configured correctly. - Useful collection API:

prefect_docker.containers.create_docker_container

,

start_docker_container

, etc. What I recommend for your case

- If the goal is “this subflow must run in a specific image,” go with pattern #1: create a Docker work pool + Docker worker, bind the subflow to that pool, and call it via

run_deployment

from your parent. This is the most reliable way to mix local and container execution in Prefect 3.x. - If you prefer all-in-one code and can tolerate local Docker calls, pattern #2 works, but you’ll need to ensure Docker access and configuration on the host where

.serve()

runs. Helpful links - How to serve flows with Docker: Serve flows with Docker - Blog (conceptual overview of running microservices with Docker + Prefect): Scalable microservices orchestration with Prefect and Docker To help further, could you share: - How you’re launching the “container tasks” today: are you using a Docker work pool/worker or calling Docker from inside tasks? - Whether you call subflows directly (function call) or via

run_deployment

. - A short error snippet/log from one failing “container” task/subflow. This will help pinpoint whether it’s an infra binding issue vs. Docker access (permissions, image, volumes, networks).

Great — you can absolutely run your flow via

.serve()

from a container and have that flow start other Docker containers. Below is a concrete way to adjust your

docker-compose.yml

and Dockerfile to meet the requirements you listed. Key changes: - Run your flow in a dedicated “runner” container that executes

.serve()

- Mount the Docker socket into that runner container so it can start other containers on the host - Install the

prefect[docker]

extra in the runner image - Use absolute host paths for any volumes you mount into containers launched by your flow - Ensure networking works for any containers your flow starts 1) Suggested docker-compose.yml - I’ve renamed your

prefect-worker

service to

prefect-runner

and changed the command to run your Python module/script that calls

.serve()

. - Added the Docker socket mount. - Gave the compose network an explicit name so you can attach spawned containers to it easily. - Left your Prefect Server service as-is.

services:
  prefect-server:
    image: prefecthq/prefect:3.2.6-python3.11-conda
    command: prefect server start --host 0.0.0.0 --port 4200
    ports:
      - "4200:4200"
    env_file:
      - .env
    environment:
      - PREFECT_SERVER_API_HOST=0.0.0.0
    volumes:
      - prefect_data:/root/.prefect
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('<http://localhost:4200/api/health>')"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    networks:
      - prefect-network

  # Container that runs your flow with .serve()
  prefect-runner:
    build:
      context: .
      dockerfile: Dockerfile
    # Replace with the entrypoint that runs your code (which calls .serve())
    command: python -m your_package.start_serve
    env_file:
      - .env
    environment:
      PREFECT_API_URL: <http://prefect-server:4200/api>
      # Optional: increase verbosity
      # PREFECT_LOGGING_LEVEL: INFO
    volumes:
      # Give the runner access to the Docker daemon
      - /var/run/docker.sock:/var/run/docker.sock
      # If your flow launches containers that need host files, mount absolute host paths:
      # - /abs/path/on/host/data:/data
    depends_on:
      prefect-server:
        condition: service_healthy
    networks:
      - prefect-network
    # To avoid socket permission issues; tighten as needed for your environment
    user: "0:0"
    restart: unless-stopped

volumes:
  prefect_data:
    name: prefect_data

networks:
  prefect-network:
    driver: bridge
    # Give it a stable, predictable name so you can connect spawned containers to it
    name: prefect-network

Notes: - If you still want a worker for other deployments, keep your worker as a separate service. If you are switching entirely to

.serve()

, you can remove the worker service. - The “runner” service can be any image that has your code and

prefect[docker]

installed. 2) Dockerfile for the runner Make sure your image installs the Docker extra so

prefect_docker

tasks work, and that it runs your code that calls

.serve()

.

# Example base — you can also use python:3.11-slim and install Prefect
FROM prefecthq/prefect:3.2.6-python3.11

WORKDIR /app
COPY . /app

# Install Prefect’s Docker extra and your dependencies
RUN pip install --no-cache-dir "prefect[docker]" \
    && pip install --no-cache-dir -r requirements.txt

# Default command can be overridden in docker-compose
CMD ["python", "-m", "your_package.start_serve"]

3) Example: flow code that runs a Docker task and serves on a schedule - This shows how to create/start a container using

prefect_docker

. - If the container you launch needs to talk to your Prefect API, you have two connectivity options: - Attach it to the same compose network by name (recommended): pass the network when creating the container so it can reach

<http://prefect-server:4200/api>

- Or set

PREFECT_API_URL=<http://host.docker.internal:4200/api>

(works well on Mac/Windows; on Linux you may need to use the host gateway IP)

from prefect import flow
from prefect_docker.containers import create_docker_container, start_docker_container

@flow
def my_flow():
    # Example: run an Alpine container
    container = create_docker_container(
        image="alpine:3.19",
        command=["sh", "-c", "echo hello from docker && sleep 1"],
        detach=True,
        name="example-task-container",
        environment={
            # Only needed if that container must reach the Prefect API
            "PREFECT_API_URL": "<http://prefect-server:4200/api>"
        },
        # Attach spawned container to the compose network so it can resolve 'prefect-server'
        # If your compose network has the explicit name set as above:
        network="prefect-network",
        # If you need to mount host paths into the spawned container, use absolute host paths:
        # volumes={"/abs/path/on/host/data": {"bind": "/data", "mode": "rw"}},
    )
    start_docker_container(container.id)

if __name__ == "__main__":
    # .serve() registers a deployment and executes scheduled runs in this process (no worker)
    # It will pause schedules on shutdown by default; disable that if you want it to resume after restarts
    my_flow.serve(
        name="my-serve",
        cron="*/5 * * * *",
        pause_on_shutdown=False
    )

4) A few gotchas to keep in mind - Permissions: mounting

/var/run/docker.sock

is necessary; running as root in the runner avoids socket permission issues. If you prefer a non-root user, add the user to the docker group (and match the group GID). - Absolute paths: when your flow spawns containers via the host Docker daemon, volume paths must be absolute paths on the host. - Networking: - By default, containers created via the Docker socket are not on your compose network. Give your compose network a stable name (

name: prefect-network

) and connect spawned containers to it (as shown) so they can reach

prefect-server

. - Alternatively, expose Prefect Server and use

<http://host.docker.internal:4200/api>

inside spawned containers (works best on Mac/Windows; on Linux you may need to use the host gateway IP). - Restarts: set

restart: unless-stopped

on the runner so

.serve()

restarts after host reboots or crashes. Consider

pause_on_shutdown=False

so your schedule resumes automatically. - What

.serve()

does in Prefect 3.x: it registers a deployment and runs scheduled flow runs within the same process/container where

.serve()

is running — no work pool or worker required. You’ll see the deployment in the UI, but the execution happens in that runner container. - More on `.serve()`: Blog: Schedule your code quickly with Flow.serve - Running flows in local processes: Run flows in local processes If you share: - the path to your flow entrypoint (so I can set the

command:

correctly), and - any host directories you need to mount into spawned containers, I can tailor the compose and flow snippets exactly to your setup.