<@ULVA73B9P> what are the ways to secure self host...
# marvin-ai
m
@Marvin what are the ways to secure self hosted prefect server. i need an user authentication ui
m
thought for 379.1 seconds
Great question. Short version: Prefect Server (self‑hosted, 3.x) does not include built‑in multi‑user auth or RBAC. It does include a simple API “basic auth” gate you can enable, and for a real login experience you typically put the Server/UI behind an auth‑enabled reverse proxy (OIDC/SAML via oauth2‑proxy, Traefik forward auth, NGINX OIDC, or AWS ALB auth). Here are the common, recommended patterns. What Prefect Server provides out of the box - API basic auth: enable a single shared credential that gates all API access. Browsers will typically show a basic‑auth prompt when the UI makes API calls. - Server env var:
PREFECT_SERVER_API_AUTH_STRING="username:password"
- Clients/workers env var:
PREFECT_API_AUTH_STRING="username:password"
- Do not set
PREFECT_API_KEY
when using self‑hosted Server (that’s for Prefect Cloud). - Useful server security settings: - Bind to localhost (and proxy from there):
PREFECT_SERVER_API_HOST=127.0.0.1
- CORS: set only your UI origin, not
*
-
PREFECT_SERVER_API_CORS_ALLOWED_ORIGINS="<https://prefect.example.com>"
- Optional base path when proxying under a subpath: -
PREFECT_SERVER_API_BASE_PATH="/prefect"
- Docs: Security settings Adding a real “user authentication UI” Because Prefect Server doesn’t have user accounts or SSO built‑in, the standard approach is to put it behind a reverse proxy that provides login (OIDC/SAML) and then forwards requests to Prefect: Option A (most common): oauth2‑proxy + NGINX/Traefik - oauth2‑proxy handles the login UI (Google Workspace, Okta, Azure AD, etc.). - The proxy: - Requires user login for the UI/API routes - After a user is authenticated, injects the Prefect Basic Auth header when proxying to the API (so the API’s
auth_string
is satisfied automatically) - High‑level flow: - Browser → NGINX (TLS, OIDC via oauth2‑proxy) → add
Authorization: Basic base64(user:pass)
→ Prefect API - Workers/CLI use
PREFECT_API_AUTH_STRING
directly to call API (no login) - oauth2‑proxy docs: https://oauth2-proxy.github.io/oauth2-proxy/ Minimal NGINX idea (conceptual)
Copy code
# TLS + OIDC handled via oauth2-proxy/forward_auth
location / {
  auth_request /oauth2/auth;
  error_page 401 = /oauth2/sign_in;

  # Inject Prefect API basic auth toward upstream after user is logged in
  proxy_set_header Authorization "Basic <base64-of-username:password>";

  proxy_pass <http://127.0.0.1:4200/;>  # or /prefect if you set BASE_PATH
}
Notes: - Keep the
username:password
identical to
PREFECT_SERVER_API_AUTH_STRING
so both human and machine clients are authorized. - Traefik and Envoy can do the same with forwardAuth and header‑injection. Option B: AWS ALB authentication (OIDC/Cognito) - ALB can front the UI/API with OIDC auth. If you go this route, it’s simpler to disable Prefect’s
auth_string
and let ALB do auth for humans. - If you still want machine clients (workers) to connect non‑interactively, consider: - Keeping a private, internal listener or bypass path for workers, or - Not using ALB auth and sticking to the oauth2‑proxy pattern above where workers just send the Basic header. - ALB OIDC auth docs: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/listener-authenticate-users.html Network hardening (fits your single‑EC2 in ap-southeast-1) - Run Prefect Server only on loopback and front it with a proxy: -
PREFECT_SERVER_API_HOST=127.0.0.1
- Start:
Copy code
export PREFECT_SERVER_API_AUTH_STRING='prefect:<long-random-secret>'
    export PREFECT_SERVER_API_HOST=127.0.0.1
    export PREFECT_SERVER_API_PORT=4200
    export PREFECT_SERVER_API_CORS_ALLOWED_ORIGINS='<https://prefect.example.com>'
    prefect server start --ui
- TLS: terminate at NGINX/Traefik or an ALB with an ACM cert. Redirect HTTP→HTTPS. - Private vs public access:
- Private only: do not assign a public IP; front with an internal ALB or just NGINX in the instance; access via VPN/Client VPN or SSM Session Manager port forwarding. - Public: assign a public IP or place behind a public ALB, lock security groups to your office/VPN IPs, add WAF if exposed to the internet, and require SSO (oauth2‑proxy/ALB auth). - RDS for Postgres: - Use TLS to RDS:
Copy code
export PREFECT_SERVER_DATABASE_CONNECTION_URL='<postgresql+psycopg://user:pass@rds-host:5432/prefect>'
    export PREFECT_SERVER_DATABASE_TLS__ENABLED=true
    export PREFECT_SERVER_DATABASE_TLS__CA_FILE=/etc/ssl/certs/rds-ca.pem
    export PREFECT_SERVER_DATABASE_TLS__CHECK_HOSTNAME=true
- Restrict the RDS SG to only the EC2’s SG; disable public access on RDS. What Prefect Server does NOT provide - No multi‑user UI, no RBAC, no API keys. Those are Prefect Cloud features. If you decide you need built‑in user management and workspace‑level permissions, Cloud is the simpler path. How this maps to your setup (single EC2 + RDS in ap-southeast-1) - Recommended starting point: 1) Keep Prefect Server bound to 127.0.0.1; use oauth2‑proxy + NGINX on the same EC2 to provide SSO login and TLS. 2) Set
PREFECT_SERVER_API_AUTH_STRING
and have NGINX inject the Basic header upstream for authenticated users. 3) Configure workers/flows on the EC2 to use
PREFECT_API_URL=<http://127.0.0.1:4200/api>
and
PREFECT_API_AUTH_STRING
so they connect locally without going through SSO. 4) Secure RDS with TLS and SG rules. 5) Decide public vs private: - Private: no public IP; access UI via VPN/SSM. - Public: ALB or public NGINX 443, restricted SG + WAF, mandatory SSO. Relevant docs and references - Prefect security settings: Security settings - Prefect Server settings (incl. base path, ports via the Helm doc but same settings apply): Server settings reference - oauth2‑proxy: oauth2-proxy - AWS ALB OIDC auth: ALB authenticate users A few questions so I can tailor a concrete config for you: - Which identity provider do you want to use (Google, Okta, Azure AD, etc.)? - Do you want the UI publicly reachable, or should it remain private (VPN/SSM only)? - Will any workers run off‑box later, or will everything stay on that single EC2? - Do you want the UI/API under a subpath (e.g., https://example.com/prefect) or its own subdomain?
m
@Marvin With these settings, the UI will prompt for the full authentication string "admin:pass" (no quotes) upon first load. It is recommended to store this information in a secure way, such as a Kubernetes Secret or in a private .env file. according to prefect ui authentication is required only first login, how to make it enable forall logins
m
thought for 130.7 seconds
Short answer: there isn’t a Prefect setting to force the UI to prompt for the API auth string on every visit. In Prefect Server (3.x), once you enable
server.api.auth_string
, the UI asks once and then caches the credential in the browser so subsequent page loads don’t prompt again. Ways to enforce re-authentication - Use a reverse proxy with real login and short-lived/session cookies (recommended) - Put the UI/API behind oauth2-proxy (or Traefik forward auth / NGINX OIDC / AWS ALB auth). - Configure it for session-only cookies (expire on browser close) or short TTL/refresh so users must re-authenticate regularly. - The proxy can inject the Prefect Basic header upstream so your API’s
auth_string
is still satisfied without users seeing “admin:pass”. - Example idea with oauth2-proxy (session cookie):
Copy code
oauth2-proxy \
      --provider=oidc \
      --oidc-issuer-url=<your-idp> \
      --client-id=<id> \
      --client-secret=<secret> \
      --cookie-secret=<32-byte-base64> \
      --cookie-secure=true \
      --cookie-expire=0          # session cookie, requires login next browser session
      --cookie-refresh=10m       # re-validate periodically
- NGINX forwards to Prefect and injects Basic auth after a user is authenticated:
Copy code
location / {
      auth_request /oauth2/auth;
      error_page 401 = /oauth2/sign_in;
      proxy_set_header Authorization "Basic <base64-of-username:password>";
      proxy_pass <http://127.0.0.1:4200/;>
    }
- Manual “logout” now - There is no built-in “logout” button in the self-hosted UI. - To force the prompt again immediately, clear site data for your Prefect UI origin (browser settings or DevTools → Application → Local Storage/IndexedDB → clear). Incognito/private windows will also prompt each time. Why this is necessary - The
auth_string
gate is a single shared Basic credential. The UI caches it client-side; there’s no server-side session or configurable “always prompt” toggle. - For multi-user login behavior, you need to front the UI/API with an auth layer (SSO proxy or ALB auth) and control session policy there. If you want, I can share a concrete NGINX + oauth2-proxy config tailored to your single-EC2-in-Singapore setup. Are you open to SSO via Google/Okta/Azure AD, and do you want the UI public or private (VPN/SSM only)? Refs: - Prefect security settings: Security settings - Server settings reference: Server settings reference - oauth2-proxy: oauth2-proxy - AWS ALB OIDC auth: ALB authenticate users
m
@Marvin I added below environment variables to ecs task, but when no user authentication asked at the first login. i attached an ALB later { "name": "PREFECT_SERVER_API_AUTH_STRING", "value": "admin:pass" }, { "name": "PREFECT_API_AUTH_STRING", "value": "admin:pass" }
m
thought for 124.9 seconds
If the UI isn’t prompting, it usually means the API isn’t actually requiring auth. Your env vars look correct, but a few common misconfigurations in ECS/ALB can prevent the prompt. Checklist to diagnose and fix 1) Verify the server actually picked up the auth setting - Exec into the ECS task container (ECS Exec) and run:
Copy code
prefect config view
You should see:
Copy code
PREFECT_SERVER_API_AUTH_STRING='****'
If it’s missing, the env var isn’t reaching the process that runs
prefect server start
. 2) Make sure you’re not accidentally pointing the UI to Prefect Cloud - If
PREFECT_API_URL
is set in the container to a Cloud URL, the UI will call Cloud (and won’t prompt). - For self-hosted Server, either unset
PREFECT_API_URL
or set it explicitly to your Server API (e.g., `https://your-alb.example.com/api`). 3) Test the API directly to confirm auth is required - Without auth (should be 401 with WWW-Authenticate):
Copy code
curl -i <https://your-alb.example.com/api/health>
- With auth (should be 200):
Copy code
curl -i -u admin:pass <https://your-alb.example.com/api/health>
If you get 200 without auth, the server is not enforcing
auth_string
. If you get 401 but no
WWW-Authenticate: Basic
header, the proxy/ALB is stripping it and the browser won’t show a prompt. 4) Confirm your ALB/proxy is passing headers and not rewriting paths - If you’re routing under a subpath, set:
Copy code
PREFECT_SERVER_API_BASE_PATH="/prefect"
and ensure the ALB forwards that path to the container unchanged. - Make sure the ALB is not removing the
WWW-Authenticate
header from 401 responses. 5) Try a private/incognito window - Browsers cache Basic credentials per-origin. If you entered them previously, you won’t be prompted again. 6) Health checks and ALB - If you enable API auth, ALB health checks to
/api/health
will return 401. Point the health check to
/
(UI static) or another public path so targets stay healthy. Known-good minimal server env (example)
Copy code
PREFECT_SERVER_API_AUTH_STRING=admin:pass
PREFECT_SERVER_API_HOST=0.0.0.0
PREFECT_SERVER_API_PORT=4200
# Optional: if you proxy under a subpath
# PREFECT_SERVER_API_BASE_PATH=/prefect
# Avoid pointing to Cloud
# unset PREFECT_API_URL
Run:
Copy code
prefect server start --ui
If you want a first-class login experience (per-user auth, repeated prompts, short-lived sessions), put the UI/API behind oauth2-proxy (OIDC/SAML) or ALB authentication and let the proxy manage sessions, then inject the
Authorization: Basic <base64(admin:pass)>
header upstream to Prefect. Details here: Security settings. If you can share: - The Prefect version in your image (
prefect version
) - The exact command your ECS task runs (entrypoint/command) - Whether
PREFECT_API_URL
is set - Your ALB listener/rule setup (path-based routing, any auth action) …I can pinpoint why the prompt isn’t appearing and provide a targeted fix.
m
@Marvin PREFECT_API_URL is not set to ALB url, can it be the reason
m
thought for 60.7 seconds
Short answer: No — not setting PREFECT_API_URL to your ALB URL is not why the UI didn’t prompt. Why: - The browser UI calls the API at the same origin (e.g., https://your-alb/api) by default. It does not use the server container’s PREFECT_API_URL. - PREFECT_API_URL only affects Prefect clients (CLI, workers, services) running inside that container, not the browser. If there’s no prompt, it means the API isn’t actually requiring auth or the 401 challenge header isn’t reaching the browser. Let’s quickly pinpoint which: 1) From inside your ECS task container, verify the server loaded the setting
Copy code
prefect config view
You should see:
Copy code
PREFECT_SERVER_API_AUTH_STRING='****'
If it’s missing, the env var didn’t reach the process that runs
prefect server start
. 2) From your workstation, test the ALB endpoint - Expect 401 with WWW-Authenticate when no credentials:
Copy code
curl -i https://<your-alb>/api/health
- Expect 200 with credentials:
Copy code
curl -i -u admin:pass https://<your-alb>/api/health
Interpretation: - 200 without creds → server is not enforcing auth (auth_string not applied). - 401 without a WWW-Authenticate: Basic header → something in front (ALB/other proxy) is stripping the header so the browser won’t show a prompt. 3) Common fixes - Ensure your task is actually starting Prefect with:
Copy code
prefect server start --ui
- Make sure nothing sets PREFECT_API_KEY in that container (that’s for Cloud and can confuse things). - Bind to all interfaces for ALB traffic:
Copy code
PREFECT_SERVER_API_HOST=0.0.0.0
PREFECT_SERVER_API_PORT=4200
- If using a subpath behind ALB, set:
Copy code
PREFECT_SERVER_API_BASE_PATH=/prefect
and make sure ALB forwards the path unchanged. - Try an incognito window to rule out cached Basic creds. 4) ALB health checks - If auth is enabled, ALB health checks to /api/health will get 401. Point health checks to “/” (the static UI) or another unauthenticated path. If you share: - The output of
prefect config view
from inside the ECS task - The result of the two curl commands above - Your ECS task command/entrypoint and ALB listener/rule setup …I can zero in on the exact cause and give you a concrete fix. Reference: Prefect security settings