<@ULVA73B9P> How can I deploy a flow with a yaml file to a self-hosted prefect server behind a nginx...
j
@Marvin How can I deploy a flow with a yaml file to a self-hosted prefect server behind a nginx reverse proxy
m
thought for 556.7 seconds
Great question — you can absolutely deploy from a YAML file to a self-hosted Prefect 3 server behind NGINX. Below is a practical, end-to-end setup you can adapt. If anything about your environment differs (e.g., you’re serving under a subpath like /prefect, or using Docker/Kubernetes workers), let me know and I’ll tailor it. Assumptions - You’re on Prefect 3.x. - You have a self-hosted Prefect Server (API + UI) reachable via an NGINX reverse proxy. - You’ll use
prefect.yaml
and the
prefect deploy
command. - A work pool exists and a worker will be running against your server. 1) Point your CLI/worker at your server (behind NGINX) Set the API URL everywhere you invoke Prefect (local dev machine, CI, worker environment) to your public, proxied URL:
Copy code
export PREFECT_API_URL="<https://your-domain.example.com/prefect/api>"
- This is the only variable your CLI/worker needs to talk to the API. - If you’re running the Prefect UI separately, set its backend target with:
Copy code
export PREFECT_UI_API_URL="<https://your-domain.example.com/prefect/api>"
so the UI knows where to make API calls. Docs: - Self-hosting Prefect Server - Security settings and environment variables 2) Example NGINX config (with subpath, websockets, and timeouts) If you’re exposing the UI at /prefect and the API at /prefect/api, here’s a minimal NGINX config that handles reverse proxying, websockets, and SPA routing:
Copy code
map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

server {
  listen 80;
  server_name <http://your-domain.example.com|your-domain.example.com>;

  # If using TLS, include your ssl_certificate/ssl_certificate_key and listen 443 ssl;

  # Prefect API (backend at <http://prefect-api:4200/api>)
  location /prefect/api/ {
    proxy_pass <http://prefect-api:4200/api/;>
    proxy_http_version 1.1;

    # Forwarded headers for correct scheme/host detection
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Port $server_port;

    # Websockets/log streaming
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    # Timeouts for long requests/streams
    proxy_read_timeout 600s;
    proxy_send_timeout 600s;
  }

  # Prefect UI (backend at <http://prefect-ui:8080>)
  location /prefect/ {
    proxy_pass <http://prefect-ui:8080/;>
    proxy_http_version 1.1;

    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Port $server_port;

    # SPA fallback so deep links work
    try_files $uri /index.html;
  }
}
Notes: - If you’re not using a subpath, drop the /prefect prefix and map
/api/
to the API and
/
to the UI instead. - Keep proxy_http_version 1.1 and the Upgrade/Connection headers to support websockets/log streaming in the UI. 3) A minimal prefect.yaml you can deploy This example deploys a flow from a GitHub repository and targets a work pool. The
pull
section is critical so your remote execution environment (e.g., Docker/Kubernetes worker) knows how to fetch the code.
Copy code
name: my-project
prefect-version: ">=3.0.0"

pull:
  - prefect.deployments.steps.git_clone:
      repository: "<https://github.com/your-org/your-repo.git>"
      branch: "main"
      # access_token: "{{ env.GH_TOKEN }}"  # if needed for private repos

deployments:
  - name: my-deployment
    entrypoint: "flows/my_flow.py:my_flow"
    work_pool:
      name: "my-pool"
      # Optional job variables for your chosen worker type
      # job_variables:
      #   image: "<http://ghcr.io/your-org/your-image:{{|ghcr.io/your-org/your-image:{{> env.GIT_SHA }}"
    parameters:
      param1: "value"
    schedules:
      - cron: "0 * * * *"
        timezone: "UTC"
Full docs on the YAML (fields like pull, deployments, schedules, work_pool):
- prefect.yaml reference - Creating deployments 4) Create a work pool and run a worker You can create the pool in the UI or via CLI:
Copy code
prefect work-pool create "my-pool" --type docker
Then start a worker that points at your pool. Make sure the worker environment also has
PREFECT_API_URL
set to the proxied URL:
Copy code
export PREFECT_API_URL="<https://your-domain.example.com/prefect/api>"
prefect worker start -p "my-pool"
5) Deploy from the YAML From your project root (where
prefect.yaml
lives): - Deploy just one deployment by name:
Copy code
prefect deploy --name "my-deployment"
- Or deploy all deployments listed in your YAML:
Copy code
prefect deploy --all
The CLI will talk to your server at
PREFECT_API_URL
and create/update the deployment(s). You’ll see them in the UI, and your worker will begin picking up flow runs based on schedules or manual triggers. Common pitfalls to watch for - 404s in the UI when served under a subpath: ensure your NGINX routes are correct, and SPA fallback (
try_files $uri /index.html
) is set for the UI location. - Missing logs/real-time updates: check the websocket headers in NGINX (`Upgrade`/`Connection`) and timeouts. - Workers not connecting: double-check
PREFECT_API_URL
everywhere (CLI, worker, CI) points to the proxied external URL, not the internal container address. Would you like me to tailor this to your exact setup? - Are you serving Prefect under a subpath like /prefect? - Which work pool type are you using (process, docker, kubernetes)? - Is your code in GitHub or bundled into an image? I can adjust the YAML and NGINX config accordingly once I have those details.