<@ULVA73B9P> Is it possible to run prefect-server ...
# marvin-ai
j
@Marvin Is it possible to run prefect-server and the user interface in a Docker container (on my machine). And on my host machine, via the CLI command, deploy flows and perform ad-hoc deployments using prefect-server, which is inside the container? In the prefect architecture, which of the Docker services is responsible only for the API and which is responsible for the interface? Please provide me with a basic compose file that allows me to test this. (I'll need to open the UI in my browser.)
m
thought for 371.3 seconds
Short answer: Yes. You can run Prefect Server (API + UI) in a Docker container and point your local CLI at it to deploy flows and trigger ad-hoc runs. Key points - In Prefect 3.x, the Prefect Server process serves both the API and the UI from a single service. There isn’t a separate “UI service” container unless you explicitly disable the UI. - By default, the UI is available at http://localhost:4200 and the API at http://localhost:4200/api. - To have your host CLI talk to the in-container server, set
PREFECT_API_URL
on your host to
<http://localhost:4200/api>
. Minimal docker-compose.yml (server + UI only)
Copy code
version: "3.9"

services:
  prefect-server:
    image: prefecthq/prefect:3-latest
    command: >
      prefect server start
      --host 0.0.0.0
      --port 4200
      --ui
    ports:
      - "4200:4200"
    volumes:
      - prefect_data:/root/.prefect
    restart: unless-stopped

volumes:
  prefect_data:
How to use it 1) Start the server:
Copy code
docker compose up -d
Open the UI at http://localhost:4200. 2) Point your host CLI at the containerized server: - One-time profile setting:
Copy code
prefect config set PREFECT_API_URL="<http://localhost:4200/api>"
- Or per-command:
Copy code
PREFECT_API_URL="<http://localhost:4200/api>" prefect deploy ...
3) Deploy and run a flow from your host: - Create a work pool (for local process execution):
Copy code
prefect work-pool create -n my-pool -t process
- Start a worker (in another terminal on your host):
Copy code
prefect worker start -p my-pool
- From your project directory, deploy your flow:
Copy code
prefect deploy ./path/to/flow.py:my_flow -n my-deployment -p my-pool
- Trigger an ad-hoc run:
Copy code
prefect deployment run "my_flow/my-deployment" --watch
Notes - The compose file above persists Prefect state to a Docker volume (
prefect_data
) so restarts don’t reset your server’s database (SQLite by default). - If port 4200 is in use, change both the container’s
--port
flag and the host port mapping, and update
PREFECT_API_URL
accordingly. - For more advanced/production setups (e.g., Postgres, reverse proxies), see the self-hosting guides. Docs - Self-host Prefect Server with Docker: Self-host with Docker - Self-host with Docker Compose: Docker Compose guide If you’d like, I can extend the compose file to include a worker service as well so everything runs inside Docker.
j
@Marvin Okay, the prefect-server in the container registered the deployment and executed the flow ad-hoc, and I could see it through the UI. However, I'd like to know if it's possible to deploy a worker in the container with some Python packages I'll need, for example: prefect-airbyte; prefect-github; prefect-email; prefect-databricks; Are they all valid and up-to-date? If I run the deployment registration in my CLI (host) by selecting the work-pool of the same worker in the container, and it's active, will it be able to execute it? And can I execute the other worker's flows ad-hoc without having to enter the container? For the scenario above, include a worker in the compose file with a volume mapped from a host folder to the app/ folder in the container (to read the file that will be executed). It should also include the installation of the necessary packages in a uv environment.
m
thought for 411.3 seconds
Great questions — here’s how to set this up cleanly with Prefect 3.x. 1) Are these packages valid and up-to-date? Yes — these are valid Prefect 3.x collections and are actively maintained. Install them by their PyPI names below: - prefect-airbyte — https://docs-3.prefect.io/integrations/prefect-airbyte/index - prefect-github — https://docs-3.prefect.io/integrations/prefect-github/index - prefect-email — https://docs-3.prefect.io/integrations/prefect-email/index - prefect-databricks — https://docs-3.prefect.io/integrations/prefect-databricks/index 2) Can a deployment registered on your host run on the worker in the container? Yes. As long as: - The deployment targets the same work pool that your containerized worker is polling - Your host CLI is pointed at the same Prefect server (PREFECT_API_URL) Then the worker in the container will pick up and execute the run. You can also trigger runs ad-hoc from your host (no need to exec into the container) using either the UI or: - List deployments:
prefect deployment ls
- Trigger a run:
prefect deployment run 'flow-name/deployment-name'
We verified these CLI commands exist in Prefect 3.x: -
prefect deploy --help
-
prefect deployment --help
-
prefect worker start --help
3) Example: add a worker to your docker-compose Below is a worker service that: - Connects to your Prefect Server container - Mounts a host folder to /app inside the worker (so your flow can read files there) - Installs the needed integrations using uv at container start, then starts the worker Assumptions: - Your Prefect Server service is named
prefect-server
and exposes 4200 - You already created a process work pool named
my-process-pool
(if not, see notes below)
Copy code
services:
  # ... your existing prefect-server service here ...

  prefect-worker:
    image: prefecthq/prefect:3-latest
    depends_on:
      - prefect-server
    environment:
      # Point the worker at your Prefect API in the server container
      PREFECT_API_URL: <http://prefect-server:4200/api>
      PREFECT_LOGGING_LEVEL: INFO
    volumes:
      # Map a host folder (adjust ./host_app) to /app in the container
      - ./host_app:/app
    # Install uv (if not present) and your integrations with uv, then start the worker
    command: >
      bash -lc "
        python -m pip install -U uv &&
        uv pip install --system
          prefect-airbyte
          prefect-github
          prefect-email
          prefect-databricks
        && prefect worker start --pool my-process-pool --name container-worker-1
      "
    restart: unless-stopped
Notes and quick commands - Create a process work pool (run on your host, pointed at the same API): -
prefect work-pool create -t process my-process-pool
- Make sure your host CLI is pointed to the same server (if you’ve published 4200): -
export PREFECT_API_URL=<http://localhost:4200/api>
- Register a deployment from your host: - From your project root:
prefect deploy
- Select
my-process-pool
when prompted (or set it in your deployment config) - Trigger runs without entering the container: - UI: open the deployment and click “Run” - CLI:
prefect deployment run 'flow-name/deployment-name'
When to use this pattern - The example above uses a process worker, which runs your flow code in the same environment as the worker. That’s ideal when you: - Need simple local file access via a volume (like /app) - Want to centrally install Python packages once in the worker container - If you switch to a Docker or Kubernetes work pool, be sure the runtime image for each flow has these integrations installed (not just the worker image). If you share your current docker-compose for the server, I can tailor the worker service exactly to your setup.