Thanks for the detailed context — that helps a lot. Short answer up front - The errors you posted are consistent with an SSL/TLS handshake being aborted mid-connection (asyncpg

start_tls

cancelled; Postgres “SSL error: unexpected EOF”). If your CNPG cluster requires or prefers TLS (very common), you should explicitly enable TLS on the Prefect server and provide the CA. That will make the handshake/verification explicit and more resilient. - For pool settings, the defaults are generally fine; if you see intermittent issues under failover or network blips, modest tuning of recycle/timeout can help. Use the

PREFECT_SERVER_DATABASE_...

variables (not the generic

PREFECT_SQLALCHEMY_...

ones) for the server. Why this is happening - The Prefect server log shows asyncio cancelling

loop.start_tls(...)

, which only occurs when the client is upgrading to TLS. Your Postgres logs at the exact moment show SSL negotiation failing/reset by peer. Together, that indicates a TLS handshake that was interrupted (e.g., brief CNPG failover, network flap, TLS secret rotation, or a proxy/load-balancer in the path). - When Prefect’s server is configured explicitly for TLS, it constructs an SSL context and passes it to asyncpg, enforcing verification and hostname checks. Without explicit TLS config, behavior can be less predictable depending on your connection URL and CNPG policies. Should you set PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED? - If your CNPG exposes TLS (recommended/typical), yes — set it to

true

and supply the CA bundle so the client validates the server properly. This tends to reduce odd “unexpected EOF” failures during handshake. - Also ensure the hostname you connect to matches the certificate’s CN/SAN. If using a service name that doesn’t match, you can set

TLS_CHECK_HOSTNAME=false

as a workaround (preferred fix is to connect using a name that matches the cert). Suggested starting config for CNPG Add these to your prefect-server Deployment (mount the CA secret from CNPG so the path exists):

# Connection URL (example)
PREFECT_SERVER_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:password@cnpg-cluster.namespace.svc.cluster.local:5432/prefect>

# TLS
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED=true
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CA_FILE=/var/run/secrets/cnpg/ca.crt
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CHECK_HOSTNAME=true
# If mTLS is required by your CNPG policy:
# PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CERT_FILE=/var/run/secrets/cnpg/tls.crt
# PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_KEY_FILE=/var/run/secrets/cnpg/tls.key

Pool settings — which to use and recommended values Use the server-scoped variables below. The generic

PREFECT_SQLALCHEMY_*

variables are not used by the Prefect 3 server. - PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE - Default: 5 - Recommendation: Leave at 5–10 unless you’ve measured persistent pool exhaustion. With two API replicas and moderate traffic, 5–10 per pod is typical. - PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE - Default: 3600 seconds - Recommendation: 600–1800 if you want to more aggressively refresh connections to avoid stale sockets across CNPG failovers/LB timeouts. A common choice is 900. - PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT - Default: 30.0 seconds - Recommendation: Consider 60 if you see timeouts acquiring a connection during brief failovers or spikes. - Also available: PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW (default 10) If you use PgBouncer in front of CNPG - Disable statement caching:

PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE=0
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_PREPARED_STATEMENT_CACHE_SIZE=0

Optional timeouts you may consider - Increase connection timeout if failovers are slightly slow: ``` PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT=5.0 # default; consider 10–15 if needed

PREFECT_SERVER_DATABASE_TIMEOUT=10.0 # statement timeout passed to asyncpg ``` Why prefer the server-scoped variables - The Prefect 3 server reads from

PREFECT_SERVER_DATABASE_*

settings. The generic

PREFECT_SQLALCHEMY_POOL_SIZE

is not used by the server process in 3.x. References (source-verified) - Server DB settings schema (TLS and pooling): prefect/settings/models/server/database.py - Async Postgres engine uses SSL context when TLS is enabled: prefect/server/database/configurations.py - Settings overview: Prefect settings docs A few clarifying questions to tune this further - Is TLS required/enabled on your CNPG cluster? If yes, do you have the CA path you can mount into the Prefect server pods? - Are you using PgBouncer/connection pooler in front of CNPG? - Do you see CNPG failovers or pod restarts around the timestamps of these errors? If you share those details (and your connection URL), I can suggest exact values for your environment.

Great questions — here’s how to think about it for CNPG + Prefect 3.x. Is TLS_ENABLED only for mTLS? - No.

PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED

applies to any TLS. - Use it for standard server-auth TLS (most CNPG setups). mTLS is only needed if your CNPG policy requires client certs — in that case you’d also set

...TLS_CERT_FILE

and

...TLS_KEY_FILE

. - CNPG often uses mTLS internally between its components; clients like Prefect typically use server-auth TLS only unless you’ve enforced client cert auth. Is your connection string OK? - Yes, this is correct:

<postgresql+asyncpg://USER:PASSWORD@HOST:5432/DBNAME>

- For TLS, Prefect passes an SSL context to asyncpg via connect_args (not the URL), so the URL is fine as-is. - Make sure

HOST

is the CNPG service name that matches the certificate’s CN/SAN (e.g., the cluster’s -rw service DNS). If it doesn’t match, you’ll need to either use the matching DNS name or set

...TLS_CHECK_HOSTNAME=false

(prefer using a matching DNS name). Is this CA path OK? - Yes — if that’s where you mount the CNPG CA into the Prefect server pods:

PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CA_FILE=/var/run/secrets/cnpg/ca.crt

- The exact path is up to you; it just needs to point at the CA file in the mounted secret. Given your answers - TLS is enabled on CNPG, and you’re not using PgBouncer. No failovers observed. - I recommend enabling TLS explicitly on Prefect and mounting the CNPG CA. This tends to eliminate the “SSL error: unexpected EOF” / asyncpg

start_tls

cancellations you’re seeing. What to set (minimal, standard server-auth TLS)

PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED=true
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CA_FILE=/var/run/secrets/cnpg/ca.crt
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CHECK_HOSTNAME=true

- Only add these if CNPG requires client certs:

PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CERT_FILE=/var/run/secrets/cnpg/tls.crt
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_KEY_FILE=/var/run/secrets/cnpg/tls.key

How to mount the CA in your Deployment (example) - Find the CNPG secret that contains `ca.crt`:

kubectl get secret -n <namespace> | grep <your-cnpg-cluster-name>
kubectl get secret <secret-name> -n <namespace> -o jsonpath='{.<http://data.ca|data.ca>\.crt}' | head

- Mount it:

volumes:
- name: cnpg-ca
  secret:
    secretName: <your-secret-with-ca.crt>

volumeMounts:
- name: cnpg-ca
  mountPath: /var/run/secrets/cnpg
  readOnly: true

Pool settings (keep defaults unless you see pressure; optional tweaks) - Use the server-scoped variables (not the generic

PREFECT_SQLALCHEMY_*

): -

PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE

(default 5) — usually fine at 5–10 -

PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE

(default 3600) — consider 600–1800 to refresh sockets more often -

PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT

(default 30.0) — consider 60 if you hit timeouts during brief blips -

PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW

(default 10) Why this helps - With TLS explicitly configured, Prefect builds a proper SSL context (min TLS 1.2,

CERT_REQUIRED

, optional hostname verification) and passes it to asyncpg. That removes ambiguity and aligns with CNPG’s TLS setup, reducing handshake/EOF resets. References (source-verified) - Server DB settings (TLS + pooling): settings/models/server/database.py - Async Postgres engine SSL handling: server/database/configurations.py - Settings overview: Prefect settings docs If you’d like, share the CNPG service DNS you use in the URL and I can confirm whether hostname checking will pass or suggest the right value for

TLS_CHECK_HOSTNAME

.

Great question — the wording on that issue is a bit misleading. What the code actually does in Prefect 3.x is:

PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED

turns on creation of an SSLContext for the asyncpg connection. That SSLContext is used for any TLS (server-auth TLS and, optionally, mutual TLS if you also provide a client cert/key). It is not limited to mTLS. Source of truth in code: - TLS settings model: SQLAlchemyTLSSettings - Engine creation (where the SSL context is built and passed to asyncpg): AsyncPostgresConfiguration.engine() Key behaviors (verified in the code): -

...TLS_ENABLED=true

just makes Prefect build an

ssl.SSLContext

and pass it to asyncpg via

connect_args["ssl"]

. - If you do not set

TLS_CERT_FILE

and

TLS_KEY_FILE

, Prefect will NOT do client cert auth — it’s just regular server-auth TLS. -

TLS_CA_FILE

is optional. If omitted, Prefect uses

ssl.create_default_context(...)

which relies on the system trust store. You only need

TLS_CA_FILE

if your DB uses a custom/enterprise CA not in the system trust. - Hostname verification is enabled by default via

TLS_CHECK_HOSTNAME=true

and certificates are required (

verify_mode = ssl.CERT_REQUIRED

) when TLS is enabled. Why you don’t need to provide a client certificate - Because client certs are only required for mTLS. If your PostgreSQL only requires server-side TLS, you can set just:

PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED=true

and omit `TLS_CERT_FILE`/`TLS_KEY_FILE`. Add

TLS_CA_FILE

only if your server’s cert is not signed by a CA your system trusts. Minimal examples: - Server-auth TLS with public CA:

PREFECT_SERVER_DATABASE_CONNECTION_URL=<postgresql+asyncpg://user:pass@host:5432/db>
  PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED=true

- Server-auth TLS with custom CA:

PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED=true
  PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CA_FILE=/etc/ssl/certs/org-root-ca.pem

- Mutual TLS:

PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED=true
  PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CA_FILE=/etc/ssl/certs/org-root-ca.pem
  PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CERT_FILE=/etc/ssl/certs/client-cert.pem
  PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_KEY_FILE=/etc/ssl/private/client-key.pem

References: - Issue proposing the settings (worded as mTLS, but the implementation supports both): #18096 - Implementation PR: #18097 If “the whole stack works most of the time” without these settings, it likely means: - Your DB is allowing non-TLS connections, or - You’re using a path that doesn’t enforce TLS, or - You haven’t been passing an SSL context at all (asyncpg won’t use TLS unless given one), but your DB doesn’t strictly require it. If you want to ensure TLS is always used and properly verified, enable

...TLS_ENABLED=true

. Only add

TLS_CA_FILE

if your system CA store can’t validate the server’s cert, and only add client `CERT`/`KEY` if your DB requires mTLS. Quick check: are you on Prefect 3.x with an asyncpg URL (

postgresql+asyncpg://...

) and does your Postgres actually require mTLS? If you can share your Postgres provider/enforcement settings, I can suggest the minimal config for your case.

Great question. Short answer: creating and passing an explicit SSLContext to asyncpg often reduces intermittent connection errors because it makes the TLS handshake predictable and compatible with your database’s expectations, instead of relying on environment-dependent defaults. Why this helps in practice: - Trust store and cert verification: In containers or minimal OS images, the default CA bundle may be missing/outdated. An explicit SSLContext lets you point to the correct CA (e.g., an AWS RDS or Cloud SQL root CA), preventing handshake failures that otherwise show up as ConnectionResetError, EOF during SSL handshake, or certificate verify failed. - Enforcing TLS parameters: You can pin minimum TLS version and acceptable ciphers to match what your DB or proxy/load balancer allows. This avoids protocol/cipher negotiation failures that manifest as “SSL: WRONG_VERSION_NUMBER” or abrupt resets. - SNI and hostname correctness: Managed Postgres (RDS, Cloud SQL, Neon, Supabase) typically expects SNI and a hostname that matches the certificate. If you connect by IP or your environment doesn’t pass SNI correctly, you’ll get cert-name mismatch or routing issues. Using the DNS hostname and an SSLContext with hostname checking ensures consistent SNI and validation. - Consistency across environments: asyncpg’s ssl=True uses system defaults; those differ across dev machines, containers, and servers. An explicit context removes that variability. - Proxies/load balancers: Some networks/load balancers require TLS or behave poorly with plaintext PostgreSQL. TLS can prevent middleboxes from interfering with the connection. A solid baseline configuration

import ssl
import asyncpg
import os

# If you’re on a managed service, download and point to their CA file.
# For AWS RDS: <https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html>
CA_PATH = "/path/to/your/root-ca.pem"

ctx = ssl.create_default_context(cafile=CA_PATH)
ctx.check_hostname = True
ctx.verify_mode = ssl.CERT_REQUIRED
# Many services require TLS 1.2+
ctx.minimum_version = ssl.TLSVersion.TLSv1_2

# Prefer using the DNS hostname (not an IP) so SNI and hostname checks work.
pool = await asyncpg.create_pool(
    host="<http://mydb.xxxxxx.rds.amazonaws.com|mydb.xxxxxx.rds.amazonaws.com>",
    port=5432,
    user="app",
    password=os.environ["PG_PASSWORD"],
    database="mydb",
    ssl=ctx,
    max_size=10,
)

Notes: - Prefer the hostname over an IP to ensure SNI and certificate hostname verification succeed. - If you don’t have a managed-service CA, you can start with

ssl.create_default_context()

and omit cafile, but using the correct CA bundle is more reliable across containers and CI. - If you still see errors after enabling SSL, the next likely culprits are network timeouts, NAT/idle connection drops, or PgBouncer configuration (especially with prepared statements). But SSLContext is a very common first fix. Could you share: - The exact error message(s) and stack trace - Where Postgres is hosted (RDS/Aurora, Cloud SQL, Neon, Supabase, self-managed) - Whether you connect by hostname or IP - asyncpg and Python versions - Whether this is running inside a Prefect flow/worker, and if you use a pool With that, I can give more targeted guidance. Helpful references: - asyncpg connection params: asyncpg.connect docs - Python SSL context: Python ssl.SSLContext - AWS RDS TLS/CA guidance: RDS SSL docs

Thanks for the update — that makes sense. Tight CPU limits can definitely manifest as intermittent 500s when the API or workers get starved. A few quick suggestions to keep things stable: - Right-size CPU requests/limits: - For self-hosted API and database: give them dedicated CPU requests and moderate limits; watch CPU throttling metrics. - For workers/executions: set realistic requests/limits in your runtime (Kubernetes, Docker, ECS). - Add autoscaling/back-pressure: - Kubernetes: enable an HPA for the API and your worker deployment; consider a conservative max concurrency for runs so you don’t saturate CPUs all at once. - Prefect-level: tune run concurrency to match available compute; add retries on tasks/flows to smooth out occasional spikes. - Monitor for early warning: - Track CPU usage vs. throttling, request latency, and 5xx rate. - Correlate spikes in scheduled/late runs with CPU saturation windows. If you’re on Kubernetes, make sure your worker/job templates include resource requests/limits. Example snippet you can drop into a base job template:

pod:
  spec:
    containers:
      - name: prefect-job
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "1"
            memory: "1Gi"

If helpful, share a bit more about your setup and I can give targeted knobs to turn: - Prefect version (assuming 3.x unless you say otherwise) - Where the CPU limit was tight (API, worker pods/containers, DB) - Orchestrator runtime (Kubernetes, Docker, ECS, etc.) and how you deploy workers - Typical peak concurrency/throughput Happy to help you dial in the exact settings.