<@ULVA73B9P> How can I connect a self hosted prefe...
# ask-marvin
j
@Marvin How can I connect a self hosted prefect on a container app connected to a flexible postgressql server with microsoft entra authentication
m
thought for 139.1 seconds
Thanks for the question — I want to make sure I understand your setup correctly before diving in: - Are you running the Prefect Server (API + UI) in Azure Container Apps? - Do you specifically need Microsoft Entra (Azure AD) authentication for Azure Database for PostgreSQL Flexible Server (i.e., no static DB password), or is standard username/password acceptable? Why I’m asking: Prefect Server (3.x) connects to Postgres via SQLAlchemy/asyncpg. It supports a static connection URL and engine options via environment variables, but it does not natively support fetching/refreshing short‑lived Azure AD access tokens for the DB connection. Azure AD tokens expire (typically ~1 hour); without a custom token-refresh mechanism at connection time, the server will eventually hit auth failures as the pool creates new connections. If a standard DB password is acceptable (recommended) - Configure the Prefect Server container with a Postgres URL and SSL. - Azure Database for PostgreSQL Flexible Server typically enforces TLS; you can either add
sslmode=require
to the URL or supply CA/cert/key via connect args. Example (using a single connection URL — simplest):
Copy code
PREFECT_SERVER_DATABASE_CONNECTION_URL=postgresql+asyncpg://<user>:<password>@<server>.<http://postgres.database.azure.com:5432/<db_name>?sslmode=require|postgres.database.azure.com:5432/<db_name>?sslmode=require>
Or with individual fields (equivalent):
Copy code
PREFECT_SERVER_DATABASE_DRIVER=postgresql+asyncpg
PREFECT_SERVER_DATABASE_HOST=<server>.<http://postgres.database.azure.com|postgres.database.azure.com>
PREFECT_SERVER_DATABASE_PORT=5432
PREFECT_SERVER_DATABASE_USER=<user>
PREFECT_SERVER_DATABASE_PASSWORD=<password>
PREFECT_SERVER_DATABASE_NAME=<db_name>
Optional TLS files (if you need to pin CA/cert/key):
Copy code
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS__TLS__CA_FILE=/path/to/ca.crt
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS__TLS__CERT_FILE=/path/to/client.crt
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS__TLS__KEY_FILE=/path/to/client.key
Example container snippet:
Copy code
services:
  prefect-server:
    image: prefecthq/prefect:3-latest
    command: prefect server start
    environment:
      PREFECT_SERVER_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:<password>@<server>.postgres.database.azure.com:5432/prefect?sslmode=require>
      # optionally tune the pool
      PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE: "10"
      PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW: "20"
    ports:
      - "4200:4200"
If you must use Microsoft Entra (Azure AD) auth - Azure AD for Postgres works by passing a short‑lived access token as the “password” for the DB user. Tokens expire frequently. - Prefect Server does not expose a hook to dynamically obtain a fresh token for each new DB connection; it expects a static URL/credentials. - Because of that, Entra token auth is not supported out of the box for Prefect’s server database. Common workarounds include: - Use a traditional Postgres user/password specifically for the Prefect database. - Place a proxy in front of Postgres that handles Azure AD token acquisition/refresh and presents stable credentials to clients (this is non-trivial; e.g., a custom sidecar or managed solution that injects tokens on each connection). - Build a custom Prefect Server image and wrapper that periodically rotates credentials and restarts the server gracefully (operationally fragile and generally not recommended). Links to Prefect docs for configuration details: - Self-hosted overview and examples: Self-hosted (Docker Compose) guide - Settings reference: Settings reference - Server database settings model (fields/environment variables): Server database config If Entra is a hard requirement, let me know: - Which identity you plan to use (Managed Identity vs Service Principal vs User)?
- Whether you can introduce a proxy/sidecar in Container Apps. - Your tolerance for a password-based DB user only for the Prefect schema. I can propose a concrete deployment pattern for your Container Apps setup once I have those details.
j
@Marvin The Microsoft Entra is a hard requirement
m
thought for 74.5 seconds
Got it — if Microsoft Entra (Azure AD) is a hard requirement, you’re covered. - Prefect Cloud 3 supports Entra ID SSO via both OIDC and SAML. - Workers/flows can authenticate to Azure resources using Entra-backed identities (Managed Identity on VMs/Container Apps, or Workload Identity on AKS) so you don’t have to store secrets. Questions so I can tailor steps: - Do you prefer OIDC (recommended for simplicity) or SAML? - Do you need group-to-role mapping or any specific claim requirements? - Is JIT provisioning sufficient, or do you need SCIM? (If SCIM is required, we should confirm your needs.) Quick setup overview OIDC with Entra (recommended) 1) In Prefect Cloud: Org Settings → Authentication → Single Sign-On → Add provider → OpenID Connect. Copy the callback/redirect URL shown there. 2) In Entra: App registrations → New registration - Redirect URI (Web) = the Prefect callback URL - Enable “ID tokens” in Authentication - Create a client secret - Optional: add “email”, “name”, and “groups” to tokens (Token configuration) 3) Back in Prefect: Enter - Issuer: https://login.microsoftonline.com/&lt;tenant-id&gt;/v2.0 - Auth URL: https://login.microsoftonline.com/&lt;tenant-id&gt;/oauth2/v2.0/authorize - Token URL: https://login.microsoftonline.com/&lt;tenant-id&gt;/oauth2/v2.0/token - JWKS URL: https://login.microsoftonline.com/&lt;tenant-id&gt;/discovery/v2.0/keys - Client ID / Client Secret - Scopes: openid profile email (plus offline_access if you require refresh tokens) - Map claims: email → email (or preferred_username), name → name, groups → groups (optional) 4) Test SSO and assign users/groups in Entra. SAML with Entra 1) In Prefect Cloud: Add SAML provider. Copy the “Entity ID” and “ACS (Reply) URL”. 2) In Entra: Enterprise applications → New application → Set up Single sign-on (SAML) - Identifier (Entity ID) = Prefect’s Entity ID - Reply URL (ACS) = Prefect’s ACS URL - NameID format = EmailAddress is typical - Map attributes: user.mail → email, user.displayname → name, groups → groups (optional) - Download IdP metadata (XML) and upload to Prefect (or paste endpoints/cert) 3) Test SSO and assign users/groups. Docs: Configure SSO in Prefect Cloud Authenticating workers/flows to Azure resources (no secrets) - Azure VM/Container Apps: use System- or User-Assigned Managed Identity. The Azure SDK will pick it up automatically with DefaultAzureCredential. - AKS: use Azure Workload Identity and run your flows via the Prefect Kubernetes worker. Annotate the service account, bind a federated identity to a user-assigned managed identity, and set AZURE_CLIENT_ID to that identity. Example flow using Entra-backed identity via Azure SDK
Copy code
from prefect import flow, task
from azure.identity import DefaultAzureCredential
from azure.storage.blob import BlobServiceClient

@task
def list_containers(account_url: str):
    cred = DefaultAzureCredential()  # picks up MI/Workload Identity automatically
    client = BlobServiceClient(account_url=account_url, credential=cred)
    return [c.name for c in client.list_containers()]

@flow
def demo():
    containers = list_containers("https://<storage-account>.<http://blob.core.windows.net|blob.core.windows.net>")
    print(containers)

if __name__ == "__main__":
    demo()
If you confirm OIDC vs SAML and any claim/group requirements, I can send a precise, tenant-ready checklist. Additionally, if you’ll be on AKS or Azure VMs, I can outline the exact identity wiring for your worker type.
j
@Marvin I am trying to deploy a self hosted prefect where the database connection points to the flexible server, I am having the issues with the authentication when the prefect server starts. Is it possible to configure the Microsoft Entra for the self hosted docker prefect?
m
thought for 313.9 seconds
Short answer: Prefect’s self-hosted (open-source) server does not have native SSO, so you can’t configure Microsoft Entra (Azure AD) directly in Prefect Server. You can still meet your Entra requirement by fronting Prefect with an auth proxy (e.g., oauth2-proxy or an ingress controller with OIDC) and by using Azure auth for your flows/workers separately. On your current issue (DB auth on startup) If the error you’re seeing is during server startup while connecting to Azure Database for PostgreSQL Flexible Server, it’s usually one of: - Wrong connection URL/driver (async required) - Missing SSL requirements for Azure - Username format mismatch (Single Server vs Flexible Server) - Accidentally using Cloud auth variables in a self-hosted setup Checklist for Azure Postgres Flexible Server - Use the async driver and include SSL:
Copy code
PREFECT_SERVER_DATABASE_CONNECTION_URL=postgresql+asyncpg://<username>:<password>@<server-name>.<http://postgres.database.azure.com:5432/<database>?sslmode=require|postgres.database.azure.com:5432/<database>?sslmode=require>
Notes: - Flexible Server typically uses just
<username>
(not
username@server
like Single Server did). - If your Azure instance requires a root CA, mount the cert in your container and add
sslrootcert=/path/to/DigiCertGlobalRootG2.crt
to the URL query. - Avoid Azure AD token-based DB authentication for the Prefect metadata DB — tokens expire and Prefect Server cannot refresh them via URL alone. Use a DB user/password. Make sure you’re not mixing Cloud vs self-hosted auth - Unset
PREFECT_API_KEY
in self-hosted environments — that’s for Prefect Cloud. - Use
PREFECT_API_AUTH_STRING
to secure access between the UI, API, workers, and clients:
Copy code
PREFECT_API_AUTH_STRING=admin:some-strong-pass
Docs: - Self-hosted with Docker Compose - Security settings - Settings reference How to meet the Entra requirement on self-hosted Since Prefect Server doesn’t do SSO natively, front it with an auth proxy that supports Azure OIDC (oauth2-proxy is a common choice). You protect the entire Prefect UI/API port behind the proxy. Example (docker-compose) with oauth2-proxy
Copy code
services:
  prefect:
    image: prefecthq/prefect:3-latest
    environment:
      PREFECT_SERVER_DATABASE_CONNECTION_URL: "postgresql+asyncpg://<username>:<password>@<server>.<http://postgres.database.azure.com:5432/<db>?sslmode=require|postgres.database.azure.com:5432/<db>?sslmode=require>"
      PREFECT_API_AUTH_STRING: "admin:some-strong-pass"
    # Prefect serves UI and API on 4200 by default
    expose:
      - "4200"
    # ... any volumes, etc.

  oauth2-proxy:
    image: <http://quay.io/oauth2-proxy/oauth2-proxy:v7.6.0|quay.io/oauth2-proxy/oauth2-proxy:v7.6.0>
    environment:
      OAUTH2_PROXY_PROVIDER: "azure"
      OAUTH2_PROXY_CLIENT_ID: "<entra-app-client-id>"
      OAUTH2_PROXY_CLIENT_SECRET: "<entra-app-client-secret>"
      OAUTH2_PROXY_COOKIE_SECRET: "<random-32-byte-base64>"
      OAUTH2_PROXY_EMAIL_DOMAINS: "*"
      OAUTH2_PROXY_SCOPE: "openid profile email"
      # Update to your public URL
      OAUTH2_PROXY_REDIRECT_URL: "<https://prefect.mycompany.com/oauth2/callback>"
      OAUTH2_PROXY_UPSTREAMS: "<http://prefect:4200>"
      OAUTH2_PROXY_HTTP_ADDRESS: "0.0.0.0:4180"
      OAUTH2_PROXY_PASS_AUTHORIZATION_HEADER: "true"
      OAUTH2_PROXY_SET_AUTHORIZATION_HEADER: "true"
      OAUTH2_PROXY_SET_XAUTHREQUEST: "true"
      OAUTH2_PROXY_WHITELIST_DOMAIN: ".<http://mycompany.com|mycompany.com>"
    ports:
      - "80:4180"
    depends_on:
      - prefect
- Point your DNS (e.g., prefect.mycompany.com) to the oauth2-proxy and let it handle Azure login, then forward to Prefect. - If you’re on Kubernetes, use NGINX Ingress with external-auth or Traefik ForwardAuth similarly. References: - oauth2-proxy Azure provider
- Cloud SSO docs (for comparison — SSO is built-in on Cloud) If you’d rather not run your own auth proxy, Prefect Cloud has native SSO with Microsoft Entra (OIDC or SAML). Next steps / to help debug your startup failure: - Can you share the exact error message from the Prefect server logs? - Which connection URL are you using now? - Are you using a DB password or attempting Azure AD token auth to Postgres? - Do you already have oauth2-proxy or an ingress in front of Prefect, or should I provide a Kubernetes ingress example? Once I have those details, I can tailor a working compose/ingress config for your setup.