Great question. Short answer: use a single container when you want the simplest, low‑maintenance setup; use Docker/Podman Compose when you need durability, scale, and production‑grade ops. A quick decision checklist - Choose single container (podman run) if: - You’re in development/evaluation, a POC, or small single‑user/team use - SQLite persistence is sufficient (no heavy concurrency, smaller metadata) - You want the fastest path to “up and running” with minimal moving parts - Choose Compose (podman-compose up) if: - You want a persistent PostgreSQL database (better concurrency and durability) - You expect multiple users, higher flow/task throughput, or longer retention - You want cleaner upgrades/backup strategies and externalized state - You may add a reverse proxy, TLS, or split services later What’s inside each - Single container - One Prefect Server container (API + UI + background services) using SQLite - Persist by mounting a volume; still a single point of failure - Easiest to run and maintain; not ideal for sustained production loads - Compose - Prefect Server container + a Postgres container (and optionally extras like a reverse proxy) - Durable state, better concurrency, and cleaner backup/restore via Postgres - Slightly more setup/ops overhead; still typically single host (for multi‑host, consider Kubernetes) RHEL9 + Podman specifics (fits your environment) - Everything below works with Podman; just replace docker with podman. For Compose, use podman-compose (or systemd units if you prefer). - If you mirror images to JFrog Artifactory, reference your Artifactory image paths in your commands/compose file. Example: simplest persistent single‑container (Podman)

podman volume create prefect-data
podman run -d --name prefect-server \
  -p 4200:4200 \
  -e PREFECT_HOME=/data \
  -v prefect-data:/data \
  <http://docker.io/prefecthq/prefect:3-latest|docker.io/prefecthq/prefect:3-latest> \
  prefect server start --host 0.0.0.0

Notes: - Persists SQLite under /data. For Artifactory, retag/push and use your registry URL instead of docker.io/prefecthq/prefect:3-latest. - Point your process worker at the server:

export PREFECT_API_URL=http://<server-host>:4200/api
  prefect worker start -p <your-work-pool>

Example: minimal Compose with Postgres (works with podman-compose)

services:
  postgres:
    image: <http://docker.io/postgres:16|docker.io/postgres:16>
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: change-me
      POSTGRES_DB: prefect
    volumes:
      - pgdata:/var/lib/postgresql/data

  prefect-server:
    image: <http://docker.io/prefecthq/prefect:3-latest|docker.io/prefecthq/prefect:3-latest>
    depends_on:
      - postgres
    ports:
      - "4200:4200"
    environment:
      PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECTION_URL: <postgresql+asyncpg://prefect:change-me@postgres:5432/prefect>
      PREFECT_SERVER_API_HOST: 0.0.0.0
    command: >
      bash -lc "prefect server start --host 0.0.0.0"

volumes:
  pgdata:

- Swap image references to your Artifactory registry as needed. - If you place the server behind a reverse proxy or different hostname, set

PREFECT_UI_API_URL

to the externally reachable API URL. Rules of thumb - Development/small team: single container with a mounted volume. - Production/multi‑user or higher throughput: Compose with Postgres. - If you later need TLS/SSO/reverse proxying, Compose makes it easier to layer those on. Docs and references - Single-container guide: Run Prefect Server in a container - Compose guide: Run Prefect Server with Docker Compose - Server concepts: Prefect Server concepts - Settings reference (database, API/UI URLs, etc.): Settings reference

If you share a bit about expected concurrency/number of users and whether you’re okay running Postgres, I can recommend a concrete path (and provide a Podman/Artifactory‑ready compose file).

Great question. Two separate concerns here: - Run containers as a specific, non-root user - Ensure a conda/mamba environment is used Before we dive in: do you want the conda environment for the Prefect Server container itself, or for a worker container that will run your flows? Most teams keep the Server simple and put conda in the worker image. I’ll show both patterns below. Run Prefect Server via Compose as a specific user (no conda needed) - Easiest: set a non-root UID/GID via the compose

user:

field and make Prefect write to a writable dir - Important: set

PREFECT_HOME

to a directory you mount and that your UID can write to - On RHEL9 with SELinux, add :Z to bind/volume mounts so labeling is correct docker-compose.yml (works with podman-compose too)

services:
  postgres:
    image: <http://docker.io/postgres:16|docker.io/postgres:16>
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: change-me
      POSTGRES_DB: prefect
    volumes:
      - pgdata:/var/lib/postgresql/data:Z

  prefect-server:
    image: <http://docker.io/prefecthq/prefect:3-latest|docker.io/prefecthq/prefect:3-latest>
    depends_on:
      - postgres
    ports:
      - "4200:4200"
    # Run as UID:GID 1001; adjust to your org’s service account UID/GID
    user: "1001:1001"
    environment:
      PREFECT_HOME: /data
      PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECTION_URL: <postgresql+asyncpg://prefect:change-me@postgres:5432/prefect>
      PREFECT_SERVER_API_HOST: 0.0.0.0
    volumes:
      - prefect-data:/data:Z
    command: >
      bash -lc "prefect server start --host 0.0.0.0"

volumes:
  pgdata:
  prefect-data:

Notes: - Using

user:

does not require the user to exist in the image; the process just runs with that numeric UID/GID. We redirect Prefect state to

/data

to avoid writing under

/root

. - Swap

<http://docker.io/|docker.io/>...

with your JFrog Artifactory image URLs. - If you prefer, you can bake a user into a custom image and drop the

user:

in compose. If you really want conda in the server container - It’s unusual (Server just needs Python deps that the official image already includes), but here’s a minimal “add micromamba + run as non-root” image. Dockerfile.server

FROM <http://docker.io/prefecthq/prefect:3-latest|docker.io/prefecthq/prefect:3-latest>

ARG UID=1001
ARG GID=1001
ARG USER=prefect

# Create a non-root user
RUN groupadd -g ${GID} ${USER} && \
    useradd -m -u ${UID} -g ${GID} ${USER}

# Install micromamba
ENV MAMBA_ROOT_PREFIX=/opt/conda
RUN apt-get update && apt-get install -y curl bzip2 && rm -rf /var/lib/apt/lists/* && \
    curl -L <https://micro.mamba.pm/api/micromamba/linux-64/latest> | tar -xvj bin/micromamba && \
    mv bin/micromamba /usr/local/bin/ && rmdir bin && \
    mkdir -p ${MAMBA_ROOT_PREFIX} && chown -R ${USER}:${USER} ${MAMBA_ROOT_PREFIX}

# Create env and put it on PATH (avoid needing activation at runtime)
USER ${USER}
SHELL ["/bin/bash", "-lc"]
RUN micromamba create -y -n server -c conda-forge python=3.11 && \
    echo 'export PATH=/opt/conda/envs/server/bin:$PATH' >> ~/.bashrc
ENV PATH=/opt/conda/envs/server/bin:$PATH

# Prefect is already installed in the base image; if you need conda-managed Prefect instead:
# RUN micromamba run -n server pip install "prefect==3.*"

ENV PREFECT_HOME=/data
WORKDIR /data

CMD ["bash", "-lc", "prefect server start --host 0.0.0.0"]

docker-compose.yml (server service only)

services:
  postgres:
    image: <http://docker.io/postgres:16|docker.io/postgres:16>
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: change-me
      POSTGRES_DB: prefect
    volumes:
      - pgdata:/var/lib/postgresql/data:Z

  prefect-server:
    build:
      context: ./server-image
      args:
        UID: 1001
        GID: 1001
        USER: prefect
    depends_on:
      - postgres
    ports:
      - "4200:4200"
    environment:
      PREFECT_HOME: /data
      PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECTION_URL: <postgresql+asyncpg://prefect:change-me@postgres:5432/prefect>
    volumes:
      - prefect-data:/data:Z

volumes:
  pgdata:
  prefect-data:

Recommended: put conda in a worker image (not the server) - This is the common pattern if you need conda for your flow dependencies. environment.yml (example)

name: flows
channels:
  - conda-forge
dependencies:
  - python=3.11
  - pip
  - pandas
  - numpy
  - pip:
      - "prefect==3.*"
      - any-private-package==x.y

Dockerfile.worker

FROM <http://docker.io/mambaorg/micromamba:1|docker.io/mambaorg/micromamba:1>

ARG UID=1001
ARG GID=1001
ARG USER=prefect

# Create a matching non-root user
RUN groupadd -g ${GID} ${USER} && useradd -m -u ${UID} -g ${GID} ${USER}
USER ${USER}

# Build the conda env
COPY environment.yml /tmp/environment.yml
# Micromamba best practice to auto-activate during RUN when requested
ENV MAMBA_DOCKERFILE_ACTIVATE=1
SHELL ["/bin/bash", "-lc"]
RUN micromamba create -y -n flows -f /tmp/environment.yml && micromamba clean --all -y

# Put the env on PATH for container start (no explicit activation needed)
ENV PATH=/opt/conda/envs/flows/bin:$PATH

# Optionally set PREFECT_HOME so the worker has a writable location
ENV PREFECT_HOME=/home/${USER}/.prefect

# Start a Prefect worker; set pool name at runtime or bake it in
CMD ["bash", "-lc", "prefect worker start -p ${PREFECT_WORK_POOL}"]

Compose with a worker

services:
  postgres:
    image: <http://docker.io/postgres:16|docker.io/postgres:16>
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: change-me
      POSTGRES_DB: prefect
    volumes:
      - pgdata:/var/lib/postgresql/data:Z

  prefect-server:
    image: <your-artifactory>/prefect-server:3
    depends_on:
      - postgres
    ports:
      - "4200:4200"
    environment:
      PREFECT_HOME: /data
      PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECTION_URL: <postgresql+asyncpg://prefect:change-me@postgres:5432/prefect>
    volumes:
      - prefect-data:/data:Z

  worker:
    image: <your-artifactory>/prefect-worker-conda:latest
    depends_on:
      - prefect-server
    environment:
      PREFECT_API_URL: <http://prefect-server:4200/api>
      PREFECT_WORK_POOL: my-process-pool
    # Optional: run as a specific UID/GID; the image already has that user created
    user: "1001:1001"
    volumes:
      - worker-data:/home/prefect/.prefect:Z

volumes:
  pgdata:
  prefect-data:
  worker-data:

RHEL9/Podman tips - Replace docker-compose with podman-compose if that’s your standard; the compose syntax above works the same - Use :Z on volumes for SELinux - If you bind-mount host paths, ensure the host path is writable by your chosen UID/GID Push/pull via JFrog Artifactory - Build locally, tag, and push:

podman build -t <your-artifactory>/prefect-server:3 ./server-image
  podman push <your-artifactory>/prefect-server:3

- Reference those images in your compose file Docs for reference - Server via Docker: Run Prefect Server in a container - Server via Compose: Run Prefect Server with Docker Compose - Settings (PREFECT_HOME, DB URL, API/UI URLs): Settings reference If you can confirm whether the conda environment is needed for Server or for a worker that runs your flows, I’ll tailor this to a minimal, Artifactory-ready compose and Dockerfile for your exact UID/GID and package needs.

Short answer: Docker Compose doesn’t change the project “type.” Pick the

prefect init

recipe based on how your flows will run, not how you host the server. Given your setup (RHEL9, process worker today, Podman available, Artifactory registry): - If you’ll keep running flows on a process worker (inside or outside Compose): - Use the minimal/default project: just run

prefect init

(no recipe). This creates a

prefect.yaml

without Docker build/push steps. - Then create deployments that target your process work pool:

prefect deploy ./path/to/flow.py:my_flow -n my-deployment -p my-process-pool

- If you plan to run flows with a Docker worker: - Use the Docker recipe:

prefect init --recipe docker

It will prompt for an image name; enter your Artifactory URL (e.g.,

<your-artifactory>/prefect/flows:{{ sha }}

). - You can also pass fields non-interactively:

prefect init --recipe docker -f image_name=<your-artifactory>/prefect/flows:{{ sha }}

- Then target a Docker work pool when deploying:

prefect deploy ./path/to/flow.py:my_flow -n my-deployment -p my-docker-pool

- Note: Prefect’s Docker worker expects a Docker-compatible API. With Podman on RHEL9, ensure the Docker socket compatibility is enabled; otherwise stick with a process worker. - If you want code pulled from Git at runtime (instead of shipping a Docker image): - You can still use the minimal/default project and define deployments with

source

(or use

flow.from_source(...).deploy(...)

). This is orthogonal to Compose. Key point: Compose is just how you run the server (and optionally workers). The

prefect init

choice maps to how your flows are packaged and executed (process vs docker). Helpful docs: - Init command: prefect init - Prefect YAML and deployments: prefect.yaml and Create deployments - Workers overview: Workers - Docker Compose server guide: Run Prefect Server with Docker Compose If you confirm whether you’ll stick with the process worker or move to a Docker worker, I can give you a minimal

prefect.yaml

template tailored to your choice (and include Artifactory image naming if you go with Docker).

Short answer: Docker Compose for the server doesn’t dictate your worker type. Choose the worker based on how you want your flows to execute. Use a Docker worker when - You want per-run isolation and fully reproducible environments - Your flows need system packages, GPUs, or complex native/OS deps - You want to build and pull versioned images from Artifactory for each deployment - You need resource controls (CPU/memory limits), custom networks, or volume mounts per run - Multiple hosts will execute flows and you want a portable, consistent runtime Use a Process worker when - You’re running on a single RHEL9 host and can manage a shared Python/conda environment - Your dependencies are simple and you don’t need container isolation - You want the simplest, fastest startup and no dependency on a container socket - You’re already happy with your current process worker reliability Podman specifics (RHEL9) - Docker worker works with Podman’s Docker-compatible API. Set DOCKER_HOST to the Podman socket: - Rootful:

export DOCKER_HOST=unix:///run/podman/podman.sock

- Rootless (typical user session):

export DOCKER_HOST=unix:///run/user/<UID>/podman/podman.sock

- Install the docker integration for Prefect when using a Docker worker:

pip install 'prefect[docker]'
  # or with uv:
  uv run --with 'prefect[docker]' prefect worker start -p my-docker-pool

Typical setup commands - Process worker

prefect work-pool create my-process-pool --type process
  prefect worker start -p my-process-pool

- Docker worker (with Podman)

prefect work-pool create my-docker-pool --type docker
  export DOCKER_HOST=unix:///run/podman/podman.sock   # or rootless path
  pip install 'prefect[docker]'
  prefect worker start -p my-docker-pool

Notes for your environment - If you keep using a process worker (your current setup), stick with the minimal/default

prefect init

and deploy to your process pool. You can manage dependencies with conda on the host. - If you move to a Docker worker, use

prefect init --recipe docker

, push images to Artifactory, and target the Docker pool in your deployments. This is the better long-term choice if you want strong isolation, consistent builds, and easy rollback. Helpful docs - Workers overview: Workers - Manage work pools: Manage work pools - Customize job variables (limits, volumes, env): Customize job variables - Docker deployment guide: Run flows with Docker If you share your flow dependency profile (pure Python vs. system libs), concurrency expectations, and whether you want to standardize on Artifactory images, I can recommend a concrete path and provide a ready-to-use

prefect.yaml

template accordingly.

There’s no hard-pinned PostgreSQL version in the Prefect 3.x docs; any modern, supported Postgres works. In practice, use a current major (15 or 16). I generally recommend postgres:16 (or a pinned minor like 16.4) for Compose. Key requirements and tips - pg_trgm extension is required. Make sure it’s enabled in your Prefect database. - Prefect uses asyncpg via SQLAlchemy; your connection URL should be postgresql+asyncpg://... - Migrations run automatically on server start by default. Compose example (with pg_trgm enabled on first init)

services:
  postgres:
    image: <http://docker.io/postgres:16|docker.io/postgres:16>
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: change-me
      POSTGRES_DB: prefect
    volumes:
      - pgdata:/var/lib/postgresql/data
      # Runs only on first init of the datadir; enables pg_trgm in the POSTGRES_DB
      - ./db-init:/docker-entrypoint-initdb.d:ro

  prefect-server:
    image: <http://docker.io/prefecthq/prefect:3-latest|docker.io/prefecthq/prefect:3-latest>
    depends_on:
      - postgres
    ports:
      - "4200:4200"
    environment:
      PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECTION_URL: <postgresql+asyncpg://prefect:change-me@postgres:5432/prefect>
      PREFECT_SERVER_API_HOST: 0.0.0.0
    command: >
      bash -lc "prefect server start --host 0.0.0.0"

volumes:
  pgdata:

Create db-init/00_extensions.sql

CREATE EXTENSION IF NOT EXISTS pg_trgm;

Notes for your environment - If you mirror images to JFrog Artifactory, swap the image references to your registry. - With Podman/podman-compose, this compose works the same. - If you already have a Postgres instance, just run:

psql -d prefect -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;"

- For large upgrades, you can raise the DB timeout if needed:

export PREFECT_API_DATABASE_TIMEOUT=600

Docs - Server concepts (pg_trgm requirement and DB settings): Prefect Server concepts - Docker Compose guide: Run Prefect Server with Docker Compose - Settings (DB URL, migrations): Settings reference If you prefer a different major (e.g., 14 or 15) due to internal standards, that’s fine—just ensure pg_trgm is enabled. Want me to tailor the compose to your Artifactory URLs and SELinux labeling for RHEL9 (:Z)?

Short answer: if your primary goals are “access host files” and “run as a specific user,” a Process worker is the simplest and most reliable choice. A Docker worker can do both, but requires extra setup (volume mounts, UID/GID alignment, Podman/Docker socket). When to pick each - Process worker (recommended for your case) - Pros: Native access to the host filesystem, just run the worker as the target user; no volume mounts or UID/GID juggling. - Cons: No container isolation; you manage dependencies on the host (e.g., conda). - Docker worker - Pros: Per-run isolation, reproducible environments, resource limits; great if you want to ship images via Artifactory. - Cons: Must mount volumes to access host files and set the container user to match host perms; with Podman you’ll also need the Docker-compatible socket configured. How to do it either way Process worker: run as a specific user with host file access - Start the worker as that user (systemd example):

# /etc/systemd/system/prefect-worker.service
[Unit]
Description=Prefect Process Worker
After=network-online.target

[Service]
User=prefect          # your service account
Group=prefect
Environment=PREFECT_API_URL=http://<server-host>:4200/api
ExecStart=/usr/bin/env bash -lc 'prefect worker start -p my-process-pool'
WorkingDirectory=/opt/prefect
Restart=always

[Install]
WantedBy=multi-user.target

- The worker will read/write host paths with the user’s native permissions. Docker worker: mount host files and run as a specific user - Ensure Podman’s Docker API is available to the worker: - Rootful:

export DOCKER_HOST=unix:///run/podman/podman.sock

- Rootless:

export DOCKER_HOST=unix:///run/user/<UID>/podman/podman.sock

- Install the docker extra and create a Docker pool:

pip install 'prefect[docker]'
prefect work-pool create my-docker-pool --type docker

- In your deployment’s prefect.yaml, set job variables to mount host paths and set the container user (match your host UID:GID). With SELinux on RHEL9, use :Z on bind mounts:

deployments:
  - name: my-deployment
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: my-docker-pool
      job_variables:
        image: <your-artifactory>/prefect/flows:{{ sha }}
        volumes:
          - /data/shared:/data/shared:Z
        container_create_kwargs:
          user: "1001:1001"          # match your host UID:GID
        env:
          SOME_VAR: value

- Start the worker (as any user that can talk to the Podman socket):

export DOCKER_HOST=unix:///run/podman/podman.sock
prefect worker start -p my-docker-pool

Rule of thumb for your environment (RHEL9, process worker today, Podman allowed) - If local filesystem access and user identity are the core requirements, stick with a Process worker and run it under the correct service account. - Move to a Docker worker if you also want containerized, reproducible runtime environments and are comfortable managing volume mounts/UIDs and the Podman socket. Docs - Workers overview: Workers - Docker worker/job variables: Customize job variables - Docker infra guide: Run flows with Docker If you share the host paths you need and the target UID/GID, I can provide a ready-to-use systemd unit (process worker) or a

prefect.yaml

snippet (docker worker) tailored to your setup.

Great question. Think of Docker Compose as “how you run the Prefect services,” while the work pool/worker type is “how your flow runs are executed.” They’re mostly orthogonal. How they interact - Work pool type is defined in Prefect Server (e.g., process, docker). It determines the job template and what the worker expects to do per flow run. - Docker Compose just runs containers. You’ll typically have: - A service for the Prefect Server - Optionally, a service for a worker that polls a specific work pool - The work pool type does not appear explicitly in compose.yml. It shows up indirectly in: - The worker service’s command:

prefect worker start -p <your-pool>

- The worker image and mounts: - Process worker: no container socket needed; may run as a specific user; mount any host paths the worker needs - Docker worker: needs Docker/Podman access (socket or DOCKER_HOST), and you configure per-run volumes/users via job variables in Prefect (not in compose) Where things live - In compose.yml: - How to start the services (server UI/API, and optionally a worker) - Network/ports, volumes for the containers, SELinux labels (:Z), container user, environment like

PREFECT_API_URL

- In Prefect (Server + prefect.yaml + CLI): - Work pool type (process vs docker) and base job template - Per-deployment job variables (image, volumes, user, env, resources for docker worker; working dir, env for process worker) -

prefect deploy

ties deployments to a work pool Two concrete patterns 1) Server + Process worker (simple host-like access) - Best when you want easy access to local files and to run as a specific user. - The worker runs as UID/GID you choose and directly sees any mounted host paths.

services:
  postgres:
    image: <http://docker.io/postgres:16|docker.io/postgres:16>
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: change-me
      POSTGRES_DB: prefect
    volumes:
      - pgdata:/var/lib/postgresql/data:Z

  prefect-server:
    image: <http://docker.io/prefecthq/prefect:3-latest|docker.io/prefecthq/prefect:3-latest>
    depends_on: [postgres]
    ports:
      - "4200:4200"
    environment:
      PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECTION_URL: <postgresql+asyncpg://prefect:change-me@postgres:5432/prefect>
      PREFECT_SERVER_API_HOST: 0.0.0.0
    command: >
      bash -lc "prefect server start --host 0.0.0.0"

  worker-process:
    image: <http://docker.io/prefecthq/prefect:3-latest|docker.io/prefecthq/prefect:3-latest>
    depends_on: [prefect-server]
    # Run as your service account; adjust UID:GID
    user: "1001:1001"
    environment:
      PREFECT_API_URL: <http://prefect-server:4200/api>
    # Mount host data you want the worker to access
    volumes:
      - /data/shared:/data/shared:Z
    command: >
      bash -lc "prefect worker start -p my-process-pool"

volumes:
  pgdata:

Notes: - Create the pool once:

prefect work-pool create my-process-pool --type process

- If you need direct host access without containerization, you can instead run the worker on the host (systemd) and skip the worker service. 2) Server + Docker worker (containerized runs) - Best when you want per-run isolation, images from Artifactory, and resource controls. - The worker needs to talk to Podman’s Docker-compatible API and you configure run-time mounts/users via job variables (stored in Prefect), not compose. ``` services: postgres: image: docker.io/postgres:16 environment: POSTGRES_USER: prefect POSTGRES_PASSWORD: change-me POSTGRES_DB: prefect volumes: - pgdata/var/lib/postgresql/dataZ prefect-server: image: docker.io/prefecthq/prefect:3-latest depends_on: [postgres] ports: - "4200:4200" environment: PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECTION_URL: postgresql+asyncpg://prefect:change-me@postgres:5432/prefect PREFECT_SERVER_API_HOST: 0.0.0.0 command: > bash -lc "prefect server start --host 0.0.0.0" worker-docker: image: docker.io/prefecthq/prefect:3-latest depends_on: [prefect-server] environment:

PREFECT_API_URL: http://prefect-server:4200/api # Point Prefect's Docker client at Podman (rootless example; set your UID) DOCKER_HOST: unix:///run/user/1001/podman/podman.sock # Optional: if you need auth for Artifactory pulls at runtime, configure registry creds in Prefect # Ensure this container user can access the Podman socket path user: "1001:1001" # If your environment uses a mounted socket file instead of DOCKER_HOST, mount it: # volumes: # - /run/user/1001/podman/podman.sock/var/run/docker.sockZ command: > bash -lc " pip install 'prefect[docker]' && prefect worker start -p my-docker-pool " volumes: pgdata:

Then, in Prefect (not compose), you set the Docker job variables (per pool or per deployment), for example in your `prefect.yaml`:

deployments: - name: my-deployment entrypoint: flows/my_flow.py:my_flow work_pool: name: my-docker-pool job_variables: image: <your-artifactory>/prefect/flows:{{ sha }} image_pull_policy: IfNotPresent # Per-run mounts into the ephemeral flow container: volumes: - /data/shared/data/sharedZ # Run the container as a specific UIDGID container_create_kwargs: user: "1001:1001" ``` Key takeaways - The work pool type is a Prefect concept; compose just runs services. - The only place it “shows up” in compose is: - The worker service command points to a pool name - For Docker workers, you add Podman/Docker access and install

prefect[docker]

- For Process workers, you might set

user:

and mount host paths - Per-run execution details (image, volumes, user) live in Prefect job variables, not the compose file. Docs - Docker Compose (server): Run Prefect Server with Docker Compose - Workers overview: Workers - Manage work pools: Manage work pools - Customize job variables: Customize job variables - Docker infra for flows: Run flows with Docker Do you want to run the worker inside Compose or keep it on the host via systemd? Based on that, I can share a minimal, RHEL9/Podman/Artifactory-ready template for your exact UID/GID and mount points.

Here’s the end-to-end path after your server is up via docker/podman-compose. 0) Point your CLI and workers at your self-hosted server - On any machine that will create deployments or run workers, set:

export PREFECT_API_URL=http://<server-host-or-dns>:4200/api

- If your worker runs inside the same compose network as the server, you can use:

PREFECT_API_URL=<http://prefect-server:4200/api>

1) Choose and create a work pool - Process worker (simple, host-like access; what you use today):

prefect work-pool create my-process-pool --type process

- Docker worker (containerized runs; requires Docker/Podman API):

prefect work-pool create my-docker-pool --type docker

2) Start a worker for that pool - Process worker (on the host or in compose):

prefect worker start -p my-process-pool

- Docker worker (with Podman):

export DOCKER_HOST=unix:///run/user/<UID>/podman/podman.sock   # or /run/podman/podman.sock
pip install 'prefect[docker]'
prefect worker start -p my-docker-pool

3) Initialize your project - If you’ll keep using a process worker:

prefect init

This creates a minimal

prefect.yaml

without image build/push steps. - If you’ll use a Docker worker and images in Artifactory:

prefect init --recipe docker -f image_name=<your-artifactory>/prefect/flows:{{ sha }}

4) Create a deployment for your flow Option A: Use the CLI (works for both worker types)

prefect deploy ./path/to/flow.py:my_flow -n my-deployment -p my-process-pool
# or for docker:
prefect deploy ./path/to/flow.py:my_flow -n my-deployment -p my-docker-pool

- You can also define the work pool in

prefect.yaml

and omit

-p

. Option B: From code (Prefect 3.x pattern)

from prefect import flow

@flow
def my_flow():
    ...

# Deploy from source (e.g., Git) without building an image locally
my_flow.from_source("<https://github.com/your/org-repo>", entrypoint="flows/etl.py:my_flow").deploy(
    name="my-deployment",
    work_pool_name="my-process-pool",   # or my-docker-pool
)

5) Trigger a run (or add a schedule)

prefect deployment run my_flow/my-deployment --watch

You can also add schedules either in

prefect.yaml

, in the UI, or via

prefect deploy

flags. Where this shows up in docker-compose.yml - The compose file is just how you run services. It won’t contain your deployments. - If you want to run a worker as a service alongside the server, add a service like: Process worker service

services:
  worker-process:
    image: <http://docker.io/prefecthq/prefect:3-latest|docker.io/prefecthq/prefect:3-latest>
    depends_on: [prefect-server]
    user: "1001:1001"                          # run as a specific user
    environment:
      PREFECT_API_URL: <http://prefect-server:4200/api>
    volumes:
      - /data/shared:/data/shared:Z            # host files accessible to flows
    command: >
      bash -lc "prefect worker start -p my-process-pool"

Docker worker service (with Podman)

services:
  worker-docker:
    image: <http://docker.io/prefecthq/prefect:3-latest|docker.io/prefecthq/prefect:3-latest>
    depends_on: [prefect-server]
    user: "1001:1001"
    environment:
      PREFECT_API_URL: <http://prefect-server:4200/api>
      DOCKER_HOST: unix:///run/user/1001/podman/podman.sock
    command: >
      bash -lc "pip install 'prefect[docker]' && prefect worker start -p my-docker-pool"

- For Docker workers, per-run container settings (image, mounts, UID/GID) are set as job variables in your deployment/work pool (in Prefect), not in compose. Docs to go deeper - Create deployments: Create deployments - Prefect YAML: prefect.yaml - Workers overview: Workers - Docker infra for flows: Run flows with Docker

If you tell me which path you’re taking (process vs docker worker) and your Artifactory image names (if docker), I’ll share a minimal

prefect.yaml

you can copy-paste.

You’re very close—this is mostly an API targeting mismatch. Key concept -

PREFECT_API_URL

is a client-side setting. It tells the Prefect CLI and workers which server to talk to. Setting it inside your server container doesn’t make your laptop or workers point to that server. - The UI shows whatever is in the server’s database. If your deployments were created against a different API (e.g., localhost, a prior server, or Cloud), they won’t appear here. What to do 1) Make sure your server is reachable - In your compose service for the server, ensure:

ports:
  - "4200:4200"
environment:
  PREFECT_SERVER_API_HOST: 0.0.0.0
# optional, only if you’re behind a proxy or need to override:
#  PREFECT_UI_API_URL: <http://ch12ldnix012:4200/api>

- From another machine:

curl <http://ch12ldnix012:4200/api/health>

You should see a healthy response. 2) Point your CLI and workers at this server - On the machine where you run prefect commands (deploy, worker), set:

export PREFECT_API_URL=<http://ch12ldnix012:4200/api>
prefect profile create self-hosted || true
prefect profile use self-hosted
prefect profile inspect

- Sanity check what the CLI sees:

prefect deployment ls

If this returns empty, your deployments were not registered to this server. 3) (Re)create deployments against this server - Re-run your deploy with the server targeted:

prefect deploy ./path/to/flow.py:my_flow -n my-deployment -p <your-work-pool>

Now the deployment will be in the server DB and visible in the UI. 4) Start a worker pointed at the same server - Process worker example:

export PREFECT_API_URL=<http://ch12ldnix012:4200/api>
prefect work-pool create my-process-pool --type process   # once
prefect worker start -p my-process-pool

- Docker worker (Podman) example:

export PREFECT_API_URL=<http://ch12ldnix012:4200/api>
export DOCKER_HOST=unix:///run/user/<UID>/podman/podman.sock
pip install 'prefect[docker]'
prefect work-pool create my-docker-pool --type docker    # once
prefect worker start -p my-docker-pool

Common gotchas - Setting

PREFECT_API_URL

only in docker-compose.yml for the server container has no effect on your laptop/worker. Set it on those machines (or services) too. - If you rebuilt/restarted the server without persistent storage (SQLite) or switched databases, prior deployments may be gone. With Postgres in Compose, they persist across restarts. - The UI is at http://ch12ldnix012:4200/, and it calls the API at /api by default. You usually don’t need to set

PREFECT_UI_API_URL

unless you’re behind a reverse proxy. Docs - Compose server guide: Run Prefect Server with Docker Compose - Settings (PREFECT_API_URL, server host/port, UI): Settings reference - Create deployments: Create deployments If you paste your server compose snippet and where you ran

prefect deploy

from, I can pinpoint exactly what to change.

That error just means your active profile doesn’t have a server URL set. Set

PREFECT_API_URL

in the profile you’re using (on the machine where you run Prefect CLI), then retry. Do this on your computer (not in the server container):

prefect profile create self-hosted   # if not created yet
prefect profile use self-hosted
prefect config set PREFECT_API_URL=<http://ch12ldnix012:4200/api>
prefect profile inspect
prefect deployment ls

- If

prefect deployment ls

is empty, your deployments were registered to a different API earlier. Re-run

prefect deploy ...

while this profile is active so they’re created in this server’s DB. Quick checks - From your machine, confirm the server is reachable:

curl <http://ch12ldnix012:4200/api/health>

- In your docker/podman-compose for the server, be sure you have:

ports:
    - "4200:4200"
  environment:
    PREFECT_SERVER_API_HOST: 0.0.0.0

You generally do not need

PREFECT_UI_API_URL

unless you’re behind a reverse proxy. Reminders -

PREFECT_API_URL

is a client/worker setting. Putting it in docker-compose.yml for the server doesn’t affect your laptop’s CLI. - You can also set it via environment variable for a single shell session:

export PREFECT_API_URL=<http://ch12ldnix012:4200/api>

Docs: Settings reference | Create deployments

That error means the profile exists but it has no

api.url

set yet. You have a few ways to fix it. Pick one: Option A: Set via env, then persist into the profile

export PREFECT_API_URL=<http://ch12ldnix012:4200/api>
prefect profile use self-hosted
prefect config set api.url=$PREFECT_API_URL
prefect config view

Option B: One-liner targeting that profile

PREFECT_PROFILE=self-hosted prefect config set api.url=<http://ch12ldnix012:4200/api>
prefect profile use self-hosted
prefect config view

Option C: Edit the profiles.toml file directly

nano ~/.prefect/profiles.toml

Add:

[profiles.self-hosted]
api.url = "<http://ch12ldnix012:4200/api>"

Then:

prefect profile use self-hosted
prefect config view

Sanity checks - From your machine:

curl <http://ch12ldnix012:4200/api/health>

- In your server compose, ensure it binds externally:

environment:
  PREFECT_SERVER_API_HOST: 0.0.0.0
ports:
  - "4200:4200"

If

prefect deployment ls

is still empty after switching, your deployments were created against a different server. Re-run your

prefect deploy ...

while this profile is active so they’re registered in this server. Docs: Settings and profiles | Settings reference