How to add authentication in selfhosted prefect? <@ULVA73B9P>

pradana adnyana

12/24/2025, 6:25 AM

How to add authentication in selfhosted prefect? @Marvin

Marvin

12/24/2025, 6:26 AM

✅ thought for 132.4 seconds

Marvin

12/24/2025, 6:28 AM

Great question. Assuming you’re on Prefect 3.x: self‑hosted Prefect supports built-in HTTP Basic auth, and for SSO/OIDC you’ll typically put a reverse proxy (e.g., NGINX/Traefik) in front of Prefect with an auth service like oauth2‑proxy. Quick start: built‑in Basic Auth (recommended minimum) - On the Prefect server (API/UI):

Copy code

export PREFECT_SERVER_API_AUTH_STRING="username:strongpassword"
# then start your server (e.g., prefect server start ...)

- On every client that talks to your server (workers, CLI, flows, automations):

Copy code

export PREFECT_API_AUTH_STRING="username:strongpassword"

Notes: - Requests must include Basic auth; health endpoints remain open. - WebSockets are protected (needed for live UI features). - There is no built‑in OIDC/SAML on self‑hosted; API keys are Cloud‑only. SSO/OIDC via reverse proxy (recommended for enterprise setups) - Put an auth proxy in front of Prefect (e.g., oauth2‑proxy) and terminate TLS at your reverse proxy (NGINX/Traefik). - Minimal NGINX + oauth2‑proxy example:

Copy code

# oauth2-proxy listens on :4180 and handles OIDC with your IdP
# NGINX protects all routes and forwards to Prefect on :4200

location = /oauth2/auth {
    proxy_pass <http://oauth2-proxy:4180;>
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

location / {
    auth_request /oauth2/auth;
    error_page 401 = /oauth2/sign_in;

    proxy_pass <http://prefect-server:4200;>

    # WebSocket support (important for UI)
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

- Traefik users can use a forwardAuth middleware pointing at oauth2‑proxy. - You can run with only the proxy auth, or keep Prefect’s Basic auth enabled for defense‑in‑depth. Recommended security settings - Enable CSRF protection on the server:

Copy code

export PREFECT_SERVER_API_CSRF_PROTECTION_ENABLED=true

- If you’re behind a proxy, set the UI -> API URL so the UI targets the public address:

Copy code

export PREFECT_UI_API_URL="<https://prefect.example.com/api>"

Docs and references - Security settings: Security settings (Prefect 3) - Self‑hosting guide: Self‑hosted Prefect - Helm (if you’re on Kubernetes): Server Helm guide - Settings reference: Settings reference Want me to tailor a config for your setup? Let me know: - Prefect version (2.x or 3.x)? - Where you’re running (Docker Compose, Kubernetes, bare VM)? - Which proxy (NGINX, Traefik) and whether you want OIDC (which IdP)?

pradana adnyana

12/24/2025, 6:30 AM

via docker compose @Marvin

Marvin

12/24/2025, 6:30 AM

✅ thought for 135.2 seconds

Marvin

12/24/2025, 6:32 AM

Got it — here’s a minimal Docker Compose setup for a self‑hosted Prefect 3 server with Basic Auth enabled. This protects both the API and UI; any clients (workers, CLI, flows) must send the same Basic credentials. Compose file (docker-compose.yml):

Copy code

version: "3.9"

services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: prefect
      POSTGRES_DB: prefect
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U prefect"]
      interval: 5s
      timeout: 5s
      retries: 10

  redis:
    image: redis:7
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping || exit 1"]
      interval: 5s
      timeout: 5s
      retries: 10

  prefect-server:
    image: prefecthq/prefect:3-latest
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      # Database
      PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>

      # Messaging (required in Prefect 3)
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: 6379

      # Security: Basic auth and CSRF
      PREFECT_SERVER_API_AUTH_STRING: ${PREFECT_AUTH_STRING}
      PREFECT_SERVER_API_CSRF_PROTECTION_ENABLED: "true"

      # Bind API to container network
      PREFECT_SERVER_API_HOST: 0.0.0.0
    command: prefect server start --host 0.0.0.0 --no-services
    ports:
      - "4200:4200"
    healthcheck:
      test: ["CMD-SHELL", "python -c 'import urllib.request as u; u.urlopen(\"<http://localhost:4200/api/health\>", timeout=1)' || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s

  prefect-services:
    image: prefecthq/prefect:3-latest
    depends_on:
      prefect-server:
        condition: service_healthy
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: 6379
    command: prefect server services start

  # Example worker that connects to your self-hosted API using Basic Auth
  prefect-worker:
    image: prefecthq/prefect:3-latest
    depends_on:
      prefect-server:
        condition: service_healthy
    environment:
      PREFECT_API_URL: <http://prefect-server:4200/api>
      PREFECT_API_AUTH_STRING: ${PREFECT_AUTH_STRING}
    command: prefect worker start --pool default
    restart: on-failure

volumes:
  postgres_data:
  redis_data:

.env (same directory as docker-compose.yml):

Copy code

# Use strong credentials; do not commit this file
PREFECT_AUTH_STRING=admin:change-me-now

How to run - Start the stack:

docker compose up -d

- Open the UI: http://localhost:4200 (you’ll be prompted for Basic auth) - Verify API health with auth:

Copy code

curl -u admin:change-me-now <http://localhost:4200/api/health>

- Ensure all clients (workers, CLI on your machine, job containers) set:

Copy code

PREFECT_API_URL=http://<host-or-service>:4200/api
PREFECT_API_AUTH_STRING=admin:change-me-now

Notes and tips - Do not set

PREFECT_API_KEY

on self-hosted; it’s for Prefect Cloud. - If you put Prefect behind a reverse proxy at a public URL, set the UI -> API URL so the in-browser UI calls the right API:

Copy code

PREFECT_UI_API_URL=<https://prefect.example.com/api>

- For SSO/OIDC, add an auth proxy (e.g., oauth2-proxy) and front with NGINX/Traefik in the same compose stack. I can share a ready-to-run oauth2-proxy + NGINX compose add‑on if you want that pattern. Docs - Security settings (Basic auth, CSRF): Prefect 3 Security settings

Marvin

12/24/2025, 6:32 AM

- Self-hosted guide: Self-hosted Prefect Want me to include an oauth2-proxy + NGINX block tailored to your IdP (e.g., Azure AD/Okta/Google)? If so, which proxy do you prefer and what’s your IdP?

pradana adnyana

12/24/2025, 6:42 AM

services: postgres: image: postgres:15 environment: POSTGRES_USER: prefect POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-prefect} POSTGRES_DB: prefect volumes: - postgres_data:/var/lib/postgresql/data healthcheck: test: ["CMD", "pg_isready", "-U", "prefect", "-d", "prefect"] interval: 10s timeout: 5s retries: 5 start_period: 15s redis: image: redis:7 volumes: - redis_data:/data healthcheck: test: ["CMD", "redis-cli", "ping"] interval: 10s timeout: 5s retries: 5 prefect-server: image: prefecthq/prefect:3-latest depends_on: postgres: condition: service_healthy redis: condition: service_healthy environment: # Connection strings PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:${POSTGRES_PASSWORD:-prefect}@postgres:5432/prefect # Messaging PREFECT_MESSAGING_BROKER: prefect_redis.messaging PREFECT_MESSAGING_CACHE: prefect_redis.messaging PREFECT_REDIS_MESSAGING_HOST: redis PREFECT_REDIS_MESSAGING_PORT: 6379 # Security & Auth PREFECT_SERVER_API_AUTH_STRING: ${PREFECT_AUTH_STRING} PREFECT_SERVER_API_CSRF_PROTECTION_ENABLED: "true" PREFECT_SERVER_API_HOST: 0.0.0.0 ports: - "4200:4200" command: prefect server start --host 0.0.0.0 --no-services prefect-services: image: prefecthq/prefect:3-latest depends_on: prefect-server: condition: service_healthy environment: PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:${POSTGRES_PASSWORD:-prefect}@postgres:5432/prefect PREFECT_MESSAGING_BROKER: prefect_redis.messaging PREFECT_MESSAGING_CACHE: prefect_redis.messaging PREFECT_REDIS_MESSAGING_HOST: redis command: prefect server services start prefect-worker: image: prefecthq/prefect:3-latest depends_on: prefect-server: condition: service_healthy environment: PREFECT_API_URL: http://prefect-server:4200/api PREFECT_API_AUTH_STRING: ${PREFECT_AUTH_STRING} command: prefect worker start --pool default restart: on-failure volumes: postgres_data: redis_data: Failure Failed to deploy a stack: compose up operation failed: dependency failed to start: container prefect-postgres-1 exited (1) why it raise above? @Marvin

Marvin

12/24/2025, 6:42 AM

✅ thought for 17.9 seconds

Marvin

12/24/2025, 6:42 AM

Thanks for the snippet — the immediate error is that your Postgres container exited (1). Let’s confirm exactly why, then fix a couple issues in your Compose. Step 1: Get the exact Postgres error

Copy code

docker compose logs --no-color postgres

Common causes: - Existing volume from a different Postgres major version (e.g., old data from 14 but image is 15) → Postgres exits with “data directory … is not compatible” or similar. - Corrupted or partially-initialized volume. - Permissions/ownership issues on the mounted data directory. If you don’t need the existing DB, reset the volume:

Copy code

docker compose down
docker volume ls | grep postgres_data
docker volume rm <yourproject>_postgres_data
docker compose up -d

If you do need the data and it was created by another version, you’ll need to upgrade with pg_upgrade or stick to the original major version. Step 2: Fix the Compose variables (these will cause failures later) You’ve got angle brackets in a few environment values that will make the URLs invalid. Replace these: - In prefect-server and prefect-services:

Copy code

# WRONG
PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect>:${POSTGRES_PASSWORD:-prefect}@postgres:5432/prefect

# RIGHT
PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:${POSTGRES_PASSWORD:-prefect}@postgres:5432/prefect>

- In prefect-worker:

Copy code

# WRONG
PREFECT_API_URL: <http://prefect-server:4200/api>

# RIGHT
PREFECT_API_URL: <http://prefect-server:4200/api>

Optional: put secrets in a .env (same folder as docker-compose.yml) and keep angle brackets out of values:

Copy code

POSTGRES_PASSWORD=your-strong-password
PREFECT_AUTH_STRING=admin:another-strong-password

Step 3: Re-run and verify

Copy code

docker compose up -d
docker compose ps
docker compose logs -f postgres

When the API is up, verify:

Copy code

curl -u admin:another-strong-password <http://localhost:4200/api/health>

If Postgres still exits - Please paste the output of

docker compose logs postgres

here. The exact message will tell us whether it’s a version mismatch, permission issue, or something else, and I’ll give you precise next steps.

pradana adnyana

12/24/2025, 6:55 AM

raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.cause prefect.exceptions.PrefectHTTPStatusError: Client error '401 Unauthorized' for url 'http://10.101.3.91:4200/api/admin/version' Response: {'exception_message': 'Unauthorized'} For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401 @Marvin, how to seth the auth in profile

Marvin

12/24/2025, 6:55 AM

✅ thought for 83.6 seconds

Marvin

12/24/2025, 6:56 AM

You’re getting 401 because your client doesn’t send Basic auth. Add the auth to your Prefect profile so every CLI/worker/flow uses it automatically. Quick fix with CLI (recommended) - Pick/create a profile and set URL + auth string:

Copy code

prefect profile create self-hosted
prefect profile use self-hosted
prefect config set PREFECT_API_URL="<http://10.101.3.91:4200/api>"
prefect config set PREFECT_API_AUTH_STRING="admin:your-password"

- Verify:

Copy code

prefect config view --show-sources

- Test:

Copy code

prefect server status   # or any command that hits the API

If you previously set a Cloud API key, unset it so it doesn’t override:

Copy code

prefect config unset PREFECT_API_KEY

Set it directly in profiles.toml - Edit ~/.prefect/profiles.toml and add:

Copy code

active = "self-hosted"

[profiles.self-hosted]
PREFECT_API_URL = "<http://10.101.3.91:4200/api>"
PREFECT_API_AUTH_STRING = "admin:your-password"

Using environment variables instead (per-process)

Copy code

export PREFECT_API_URL="<http://10.101.3.91:4200/api>"
export PREFECT_API_AUTH_STRING="admin:your-password"

Common pitfalls - Don’t set

PREFECT_API_KEY

for self-hosted; it’s Cloud-only and will take precedence over

PREFECT_API_AUTH_STRING

. - Make sure the server side is set with the same credentials:

Copy code

PREFECT_SERVER_API_AUTH_STRING=admin:your-password

- If you access the UI via a different host (proxy), set UI to point at that API:

Copy code

PREFECT_UI_API_URL="<https://your-domain.example.com/api>"

If it still 401s, try: -

curl -u admin:your-password <http://10.101.3.91:4200/api/health>

prefect config view --show-secrets

to confirm the auth string is loaded in the active profile.

159 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.