<@ULVA73B9P> I cannot configure the whole postgres...
# ask-marvin
u
@Marvin I cannot configure the whole postgres connection string, is there a way to provide multiple environment variables to connect to postgres
m
thought for 213.6 seconds
Yes. In Prefect 3.x you can configure Postgres without a single URL by using the prefect-sqlalchemy collection’s
ConnectionComponents
and feed each part from environment variables. Here’s a complete example that: - Builds a
SqlAlchemyConnector
from separate env vars - Saves it as a block - Uses it in a flow
Copy code
import os
from pydantic import SecretStr
from prefect import flow
from prefect_sqlalchemy.database import SqlAlchemyConnector
from prefect_sqlalchemy.credentials import ConnectionComponents

# Build the connector from discrete env vars
connector = SqlAlchemyConnector(
    connection_info=ConnectionComponents(
        # Choose the driver that matches what you have installed:
        # - "postgresql+psycopg" for psycopg3
        # - "postgresql+psycopg2" for psycopg2
        driver=os.getenv("PGDRIVER", "postgresql+psycopg"),
        host=os.environ["PGHOST"],
        port=int(os.getenv("PGPORT", "5432")),
        username=os.environ["PGUSER"],
        password=SecretStr(os.environ["PGPASSWORD"]),
        database=os.environ["PGDATABASE"],
        # Optional extra params like SSL settings
        query={"sslmode": os.getenv("PGSSLMODE", "prefer")},
    ),
    # Optional: driver-specific connect args (e.g., for SSL certs)
    # connect_args={"sslrootcert": os.getenv("PGSSLROOTCERT")},
)

# Save as a Prefect block for reuse (optional)
connector.save("pg-connector", overwrite=True)

@flow
def demo():
    # Load and use the connector in your flow
    with SqlAlchemyConnector.load("pg-connector") as db:
        db.execute("CREATE TABLE IF NOT EXISTS demo (id INT)")
        db.execute("INSERT INTO demo (id) VALUES (:id)", {"id": 1})
        rows = db.fetch_many("SELECT * FROM demo")
        print(rows)

if __name__ == "__main__":
    demo()
Notes: - Use
PGDRIVER=postgresql+psycopg
if you’re on psycopg3 (recommended). If you use psycopg2, set
PGDRIVER=postgresql+psycopg2
. Make sure the corresponding package is installed. - You can add more connection details via
query
(e.g.,
sslmode=require
) or
connect_args
if your driver needs them. Docs: - Prefect + SQLAlchemy collection: prefect-sqlalchemy docs Does this match your setup? If you need to pass additional SSL parameters (certs, keys, etc.), let me know which ones and I can show where to place them.
u
@Marvin I mean for prefect-server deployment
m
thought for 164.2 seconds
Got it — for a Prefect Server deployment you can provide Postgres settings as separate environment variables. You do not have to use a single connection URL. Supported split settings (Prefect 3.x): - PREFECT_SERVER_DATABASE_DRIVER (e.g. postgresql+asyncpg) - PREFECT_SERVER_DATABASE_HOST - PREFECT_SERVER_DATABASE_PORT - PREFECT_SERVER_DATABASE_USER - PREFECT_SERVER_DATABASE_PASSWORD - PREFECT_SERVER_DATABASE_NAME - Optional extras: - PREFECT_SERVER_DATABASE_ECHO=true|false - PREFECT_SERVER_DATABASE_MIGRATE_ON_START=true|false - PREFECT_SERVER_DATABASE_TIMEOUT=10 - PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT=5 SQLAlchemy connect args and TLS (optional): - PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_APPLICATION_NAME - PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE - PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_PREPARED_STATEMENT_CACHE_SIZE - TLS: - PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED=true|false - PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CA_FILE=/path/ca.pem - PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CERT_FILE=/path/tls.crt - PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_KEY_FILE=/path/tls.key - PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CHECK_HOSTNAME=true|false Notes - If you set PREFECT_SERVER_DATABASE_CONNECTION_URL, it will be used instead of the split fields. Use one approach or the other. - For Postgres, set the driver to
postgresql+asyncpg
. Examples Shell env
Copy code
export PREFECT_SERVER_DATABASE_DRIVER=postgresql+asyncpg
export PREFECT_SERVER_DATABASE_HOST=<http://db.example.com|db.example.com>
export PREFECT_SERVER_DATABASE_PORT=5432
export PREFECT_SERVER_DATABASE_USER=prefect
export PREFECT_SERVER_DATABASE_PASSWORD=supersecret
export PREFECT_SERVER_DATABASE_NAME=prefect
export PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_APPLICATION_NAME=prefect-server
export PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED=true
Docker Compose (server container)
Copy code
services:
  prefect-server:
    image: prefecthq/prefect:3-latest
    command: prefect server start
    environment:
      PREFECT_SERVER_DATABASE_DRIVER: postgresql+asyncpg
      PREFECT_SERVER_DATABASE_HOST: ${PGHOST}
      PREFECT_SERVER_DATABASE_PORT: ${PGPORT}
      PREFECT_SERVER_DATABASE_USER: ${PGUSER}
      PREFECT_SERVER_DATABASE_PASSWORD: ${PGPASSWORD}
      PREFECT_SERVER_DATABASE_NAME: ${PGDATABASE}
      PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_APPLICATION_NAME: prefect-server
      PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED: "true"
Kubernetes Pod env (snippet)
Copy code
env:
  - name: PREFECT_SERVER_DATABASE_DRIVER
    value: postgresql+asyncpg
  - name: PREFECT_SERVER_DATABASE_HOST
    valueFrom:
      secretKeyRef: { name: pg-secrets, key: host }
  - name: PREFECT_SERVER_DATABASE_PORT
    value: "5432"
  - name: PREFECT_SERVER_DATABASE_USER
    valueFrom:
      secretKeyRef: { name: pg-secrets, key: user }
  - name: PREFECT_SERVER_DATABASE_PASSWORD
    valueFrom:
      secretKeyRef: { name: pg-secrets, key: password }
  - name: PREFECT_SERVER_DATABASE_NAME
    value: prefect
Docs - Settings reference (search for “ServerDatabaseSettings” and “SQLAlchemyConnectArgsSettings”): Prefect 3 settings reference How are you deploying Prefect Server (Docker Compose, Kubernetes/Helm, bare VM)? I can tailor an example to your setup, including TLS/SSL if you need it.
n
@Marvin i don’t believe that’s accurate, why are you saying this?
m
thought for 80.8 seconds
You’re right to ask — let me clarify and cite sources. Why I said that: - In Prefect 3.x, the server DB config is defined by the ServerDatabaseSettings model, which has fields for
driver
,
host
,
port
,
user
,
password
, and
name
in addition to
connection_url
. Prefect’s settings system maps these nested fields to environment variables by uppercasing and prefixing with
PREFECT_
, so you can set split env vars like
PREFECT_SERVER_DATABASE_HOST
, etc. - You can see these fields in the code here: ServerDatabaseSettings in code - The general rules for settings/env mapping are described in the settings reference: Prefect 3 settings reference Practically, this means you can set: -
PREFECT_SERVER_DATABASE_DRIVER
(e.g.
postgresql+asyncpg
) -
PREFECT_SERVER_DATABASE_HOST
-
PREFECT_SERVER_DATABASE_PORT
-
PREFECT_SERVER_DATABASE_USER
-
PREFECT_SERVER_DATABASE_PASSWORD
-
PREFECT_SERVER_DATABASE_NAME
Optional SQLAlchemy/TLS extras exist under the nested connect-args model (e.g.
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED
, etc.), also visible in the same file. Notes: - If you set
PREFECT_SERVER_DATABASE_CONNECTION_URL
, that single URL will be used. The split fields are there specifically for cases where you can’t provide a full URL. - Use
postgresql+asyncpg
as the driver for Postgres. If you try the split env vars and still see Prefect not picking them up, can you share: - Your Prefect version (
prefect version
) - How you’re running Server (Docker Compose, K8s, etc.) - A quick snippet of the env you’re setting I’ll reproduce and get you an exact fix.
n
oh yep forgot about that, marvin is right
u
yeah, but the prefect server helm chart kind of madates to either create a secret with connection string or define an already existing secret, so the env value thing does not work. May be I will create a PR to helm chart
b
@Utkarsh Jain I'm assuming you're using an external Postgres instance and not the bundled one in the chart then?
I went ahead and added an issue. Feel free to add more context if you want: https://github.com/PrefectHQ/prefect-helm/issues/528
u
hey, this is exactly what I was looking for. Happy to create a PR for this
🙌 1