Cullen Baker
10/28/2025, 3:51 PMMarvin
10/28/2025, 3:51 PMMarvin
10/28/2025, 3:56 PMPREFECT_API_DATABASE_CONNECTION_URL (asyncpg DSN) and that migrations-on-start is enabled.
- Prefect defaults to migrating on start, but it’s safe to set it explicitly.
Example docker-compose snippet:
services:
postgres:
image: postgres:15
environment:
POSTGRES_DB: prefect
POSTGRES_USER: prefect
POSTGRES_PASSWORD: prefect
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 5s
timeout: 5s
retries: 10
volumes:
# Enable pg_trgm on first init
- ./postgres-init:/docker-entrypoint-initdb.d
prefect-server:
image: prefecthq/prefect:3-latest
command: ["prefect", "server", "start", "--ui"]
environment:
# asyncpg DSN is required
PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
PREFECT_API_DATABASE_MIGRATE_ON_START: "true"
# Optional if migrations or indexing take longer on first run
PREFECT_API_DATABASE_TIMEOUT: "600"
PREFECT_API_DATABASE_CONNECTION_TIMEOUT: "30"
depends_on:
postgres:
condition: service_healthy
Also add an init script to enable the required pg_trgm extension:
- Create a file ./postgres-init/01-extensions.sql with:
CREATE EXTENSION IF NOT EXISTS pg_trgm;
Option B — Run a one-time migration job before starting the server
- Run the Prefect CLI migration inside the Prefect image so it uses the same version as your server.
- This is helpful if you want explicit control or if you disabled migrate-on-start.
One-off run:
docker compose run --rm \
-e PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://prefect:prefect@postgres:5432/prefect> \
prefect-server \
prefect server database upgrade -y
Or as a dedicated init service:
services:
postgres: {...same as above...}
prefect-db-migrate:
image: prefecthq/prefect:3-latest
environment:
PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
PREFECT_API_DATABASE_TIMEOUT: "600"
command: ["prefect", "server", "database", "upgrade", "-y"]
restart: "no"
depends_on:
postgres:
condition: service_healthy
prefect-server:
image: prefecthq/prefect:3-latest
command: ["prefect", "server", "start", "--ui"]
environment:
PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
depends_on:
postgres:
condition: service_healthy
# If your compose supports it, wait for successful completion of the migration job
# prefect-db-migrate:
# condition: service_completed_successfully
Common pitfalls that cause “relation does not exist”
- Wrong connection string format: you must use the async driver scheme: <postgresql+asyncpg://user:pass@host:5432/db>
- DB not ready yet: add a Postgres healthcheck and depends_on to delay Prefect startup until Postgres is accepting connections
- Missing pg_trgm extension: ensure you run CREATE EXTENSION pg_trgm; in the target database (init script above)
- Migrate-on-start disabled: either set PREFECT_API_DATABASE_MIGRATE_ON_START=true or run prefect server database upgrade -y manually
Docs for reference:
- Docker Compose guide: Prefect 3 - Docker Compose
- Server CLI (database upgrade/downgrade): Server CLI
- Settings (DB env vars): Settings referenceMarvin
10/28/2025, 3:56 PMCullen Baker
10/28/2025, 6:14 PMMarvin
10/28/2025, 6:14 PMCullen Baker
10/28/2025, 6:15 PMservices:
postgres:
image: <http://artifactory.company.com/docker-all/postgres:16|artifactory.company.com/docker-all/postgres:16>
environment:
POSTGRES_USER: prefect
POSTGRES_PASSWORD: prefect
POSTGRES_DB: prefect
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U prefect"]
interval: 5s
timeout: 5s
retries: 5
redis:
image: <http://artifactory.company.com/docker-all/redis:7|artifactory.company.com/docker-all/redis:7>
volumes:
- redis_data:/data
healthcheck:
test: ["CMD-SHELL", "redis-cli ping"]
interval: 5s
timeout: 5s
retries: 5
prefect-server:
image: <http://artifactory.company.com/docker-all/prefecthq/prefect:3-latest|artifactory.company.com/docker-all/prefecthq/prefect:3-latest>
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
environment:
PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
PREFECT_SERVER_API_HOST: 0.0.0.0
PREFECT_SERVER_API_PORT: 4200
PREFECT_UI_URL: http://${HOSTNAME:-localhost}:4200
PREFECT_UI_API_URL: http://${HOSTNAME:-localhost}:4200/api
PREFECT_MESSAGING_BROKER: prefect_redis.messaging
PREFECT_MESSAGING_CACHE: prefect_redis.messaging
PREFECT_REDIS_MESSAGING_HOST: redis
PREFECT_REDIS_MESSAGING_PORT: 6379
PREFECT_REDIS_MESSAGING_DB: 0
command: sh -c "prefect server database upgrade --yes && prefect server start --no-services"
ports:
- "4200:4200"
healthcheck:
test: ["CMD-SHELL", "curl -f <http://localhost:4200/api/ready> || exit 1"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
prefect-services:
image: <http://artifactory.company.com/docker-all/prefecthq/prefect:3-latest|artifactory.company.com/docker-all/prefecthq/prefect:3-latest>
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
environment:
PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
PREFECT_MESSAGING_BROKER: prefect_redis.messaging
PREFECT_MESSAGING_CACHE: prefect_redis.messaging
PREFECT_REDIS_MESSAGING_HOST: redis
PREFECT_REDIS_MESSAGING_PORT: 6379
PREFECT_REDIS_MESSAGING_DB: 0
command: prefect server services start
prefect-worker:
build:
context: /company/bundles/maestro/linux64/${VERSION}
dockerfile: /company/prod/fpa/fpa_maestro_client/maestro_client/conf/Dockerfile
depends_on:
prefect-server:
condition: service_healthy
postgres:
condition: service_healthy
redis:
condition: service_healthy
environment:
# Prefect Configuration - use localhost since we're in host network mode
PREFECT_API_URL: <http://localhost:4200/api>
PREFECT_LOGGING_LEVEL: DEBUG
# Slack service URL - use actual hostname that works from host
SLACK_SERVICE_URL: <http://ptt-slack:4509/post>
# Vault credentials from environment variables
DECRYPT_VAULT: ${DECRYPT_VAULT}
PROD_APPROLE_TBM_ROLE_ID: ${PROD_APPROLE_TBM_ROLE_ID}
PROD_APPROLE_TBM_SECRET_ID: ${PROD_APPROLE_TBM_SECRET_ID}
PROD_APPROLE_ROLE_ID: ${PROD_APPROLE_ROLE_ID}
PROD_APPROLE_SECRET_ID: ${PROD_APPROLE_SECRET_ID}
PROD_APPROLE_HCA_ROLE_ID: ${PROD_APPROLE_HCA_ROLE_ID}
PROD_APPROLE_HCA_SECRET_ID: ${PROD_APPROLE_HCA_SECRET_ID}
volumes:
# NFSv3 mount points - specific directories only
- /remote/finance:/remote/finance:rw
- /remote/tbm:/remote/tbm:rw
- /remote/applications:/remote/applications:rw
# Cache directory for writable storage
- maestro_cache:/app/src/maestro/cache:rw
# Mount prefect.yaml from runtime config location
- /company/prod/fpa/fpa_maestro_client/maestro_client/conf/prefect.yaml:/app/src/maestro/prefect.yaml:ro
network_mode: host
restart: unless-stopped
volumes:
postgres_data:
redis_data:
maestro_cache:Marvin
10/28/2025, 6:15 PMMarvin
10/28/2025, 6:18 PMPREFECT_API_URL set and it includes the /api suffix, e.g. <http://prefect-server:4200/api>. Missing /api will cause readiness checks to fail.
- Service hostname: In Docker Compose, use the server service name as the host (e.g., prefect-server), not localhost.
- Server bind address: Ensure the server listens on all interfaces so other containers can reach it: prefect server start --host 0.0.0.0.
- Health + dependency: Add a healthcheck on the server using /api/ready and have the worker depend on the server being healthy.
- DB ready/migrated: Confirm Postgres is ready and migrations have run (auto-migrate on start or a one-time prefect server database upgrade -y). Also ensure pg_trgm is enabled.
Minimal example
services:
postgres:
image: postgres:15
environment:
POSTGRES_DB: prefect
POSTGRES_USER: prefect
POSTGRES_PASSWORD: prefect
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 5s
timeout: 5s
retries: 10
volumes:
# Enables pg_trgm on first init
- ./postgres-init:/docker-entrypoint-initdb.d
prefect-server:
image: prefecthq/prefect:3-latest
command: ["prefect", "server", "start", "--ui", "--host", "0.0.0.0"]
environment:
PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
PREFECT_API_DATABASE_MIGRATE_ON_START: "true"
# Optional for slower environments / first run
PREFECT_API_DATABASE_TIMEOUT: "600"
ports:
- "4200:4200"
depends_on:
postgres:
condition: service_healthy
healthcheck:
test: ["CMD", "curl", "-fsS", "<http://localhost:4200/api/ready>"]
interval: 5s
timeout: 5s
retries: 20
prefect-worker:
image: prefecthq/prefect:3-latest
command: ["prefect", "worker", "start", "-p", "my-pool"]
environment:
# Critical: include the /api suffix
PREFECT_API_URL: <http://prefect-server:4200/api>
depends_on:
prefect-server:
condition: service_healthy
Quick diagnostics
- From the worker container, try: curl -v <http://prefect-server:4200/api/ready>
- If this fails, it’s either the wrong host/port or the server isn’t listening on 0.0.0.0.
- If it 404s, your PREFECT_API_URL likely lacks the /api suffix.
- If it times out, check server healthcheck logs and Postgres readiness/migrations.
Relevant docs
- Docker Compose guide: Prefect 3 - Docker Compose
- Server CLI (including database upgrade): Server CLI
- Settings reference: Settings reference
- Server health endpoints: /api/ready and /api/health
Please paste your docker-compose.yml and I’ll point out the exact changes needed.Marvin
10/28/2025, 6:18 PMnetwork_mode: host, localhost inside the worker resolves to the host’s loopback. If your system prefers IPv6 for localhost (i.e., ::1), it can fail to reach the API if the port is only bound on IPv4.
- Quick fix: change the worker’s PREFECT_API_URL to use IPv4 explicitly:
PREFECT_API_URL: <http://127.0.0.1:4200/api>
Alternatively, remove network_mode: host, keep everything on the default Docker network, and set:
PREFECT_API_URL: <http://prefect-server:4200/api>
2) Angle brackets around URLs/DSNs in env
- In the snippet, I see <...> around URLs, e.g.:
PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
PREFECT_API_URL: <http://localhost:4200/api>
If those angle brackets are literally in your compose file, Prefect will get invalid URLs and will never connect. Make sure the env values are plain strings without `<...>`:
PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
PREFECT_API_URL: <http://127.0.0.1:4200/api>
3) UI URL envs
- Not directly related to the worker, but these affect the UI:
PREFECT_UI_URL: http://${HOSTNAME:-localhost}:4200
PREFECT_UI_API_URL: http://${HOSTNAME:-localhost}:4200/api
Inside the container, HOSTNAME is the container ID and won’t be reachable from your browser. Use a hostname your browser can reach (e.g., the machine’s DNS name or IP). For local testing:
PREFECT_UI_URL: <http://localhost:4200>
PREFECT_UI_API_URL: <http://localhost:4200/api>
4) Verify server readiness from the worker container
- This will immediately tell us if it’s a reachability/URL issue:
docker compose exec prefect-worker sh -lc 'curl -v <http://127.0.0.1:4200/api/ready>'
If that fails, try from the host itself:
curl -v <http://127.0.0.1:4200/api/ready>
You should see a 200 when the API is ready. Docs on the endpoint: Perform Readiness Check
Minimal changes I recommend right now
- Remove angle brackets from URLs/DSNs
- Pin the worker API URL to IPv4 or move it to the same Docker network as the server
Example adjusted snippet:
prefect-server:
...
environment:
PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
PREFECT_SERVER_API_HOST: 0.0.0.0
PREFECT_SERVER_API_PORT: 4200
PREFECT_UI_URL: <http://localhost:4200>
PREFECT_UI_API_URL: <http://localhost:4200/api>
PREFECT_MESSAGING_BROKER: prefect_redis.messaging
PREFECT_MESSAGING_CACHE: prefect_redis.messaging
PREFECT_REDIS_MESSAGING_HOST: redis
PREFECT_REDIS_MESSAGING_PORT: 6379
PREFECT_REDIS_MESSAGING_DB: 0
prefect-worker:
...
# Option A: keep host networking and force IPv4 loopback
network_mode: host
environment:
PREFECT_API_URL: <http://127.0.0.1:4200/api>
PREFECT_LOGGING_LEVEL: DEBUG
# Option B: recommended — remove host networking and use service DNS
# (comment out network_mode)
# environment:
# PREFECT_API_URL: <http://prefect-server:4200/api>
If this doesn’t resolve it, please:
- Confirm whether the <...> brackets are actually present in your compose file
- Share the output of:
- docker compose logs -f prefect-server
- docker compose exec prefect-worker sh -lc 'env | grep PREFECT_API_URL && curl -sv <http://127.0.0.1:4200/api/ready>'
Relevant docs:
- Docker Compose guide: Prefect 3 - Docker ComposeMarvin
10/28/2025, 6:18 PM