<@ULVA73B9P> with the docker-compose deployment, h...
# ask-marvin
c
@Marvin with the docker-compose deployment, how to initialize the prefect posgres DB? I'm getting errors like this: ERROR: relation "flow_run" does not exist ERROR: relation "deployment" does not exist ERROR: relation "configuration" does not exist
m
thought for 326.7 seconds
Those “relation does not exist” errors mean the Prefect database hasn’t been migrated yet. In Prefect 3.x, you have two supported ways to initialize (run migrations) against Postgres when using Docker Compose: Option A — Let the server migrate automatically on startup - Ensure the server container has a valid
PREFECT_API_DATABASE_CONNECTION_URL
(asyncpg DSN) and that migrations-on-start is enabled. - Prefect defaults to migrating on start, but it’s safe to set it explicitly. Example docker-compose snippet:
Copy code
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: prefect
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: prefect
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 5s
      retries: 10
    volumes:
      # Enable pg_trgm on first init
      - ./postgres-init:/docker-entrypoint-initdb.d

  prefect-server:
    image: prefecthq/prefect:3-latest
    command: ["prefect", "server", "start", "--ui"]
    environment:
      # asyncpg DSN is required
      PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
      PREFECT_API_DATABASE_MIGRATE_ON_START: "true"
      # Optional if migrations or indexing take longer on first run
      PREFECT_API_DATABASE_TIMEOUT: "600"
      PREFECT_API_DATABASE_CONNECTION_TIMEOUT: "30"
    depends_on:
      postgres:
        condition: service_healthy
Also add an init script to enable the required pg_trgm extension: - Create a file
./postgres-init/01-extensions.sql
with:
Copy code
CREATE EXTENSION IF NOT EXISTS pg_trgm;
Option B — Run a one-time migration job before starting the server - Run the Prefect CLI migration inside the Prefect image so it uses the same version as your server. - This is helpful if you want explicit control or if you disabled migrate-on-start. One-off run:
Copy code
docker compose run --rm \
  -e PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://prefect:prefect@postgres:5432/prefect> \
  prefect-server \
  prefect server database upgrade -y
Or as a dedicated init service:
Copy code
services:
  postgres: {...same as above...}

  prefect-db-migrate:
    image: prefecthq/prefect:3-latest
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
      PREFECT_API_DATABASE_TIMEOUT: "600"
    command: ["prefect", "server", "database", "upgrade", "-y"]
    restart: "no"
    depends_on:
      postgres:
        condition: service_healthy

  prefect-server:
    image: prefecthq/prefect:3-latest
    command: ["prefect", "server", "start", "--ui"]
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
    depends_on:
      postgres:
        condition: service_healthy
      # If your compose supports it, wait for successful completion of the migration job
      # prefect-db-migrate:
      #   condition: service_completed_successfully
Common pitfalls that cause “relation does not exist” - Wrong connection string format: you must use the async driver scheme:
<postgresql+asyncpg://user:pass@host:5432/db>
- DB not ready yet: add a Postgres healthcheck and
depends_on
to delay Prefect startup until Postgres is accepting connections - Missing pg_trgm extension: ensure you run
CREATE EXTENSION pg_trgm;
in the target database (init script above) - Migrate-on-start disabled: either set
PREFECT_API_DATABASE_MIGRATE_ON_START=true
or run
prefect server database upgrade -y
manually Docs for reference: - Docker Compose guide: Prefect 3 - Docker Compose - Server CLI (database upgrade/downgrade): Server CLI - Settings (DB env vars): Settings reference
- Server concepts (pg_trgm requirement): Server concepts If you can share your docker-compose.yml (redacting secrets), I can point out the minimal changes needed for your setup.
c
@Marvin Now in this docker-compose deployment I am getting this error: [prefect-worker] | Attempt 18/30 - Server not ready yet... [prefect-worker] | Attempt 19/30 - Server not ready yet... [prefect-worker] | Attempt 20/30 - Server not ready yet... [prefect-worker] | Attempt 21/30 - Server not ready yet... [prefect-worker] | Attempt 22/30 - Server not ready yet... [prefect-worker] | Attempt 23/30 - Server not ready yet... [prefect-worker] | Attempt 24/30 - Server not ready yet... [prefect-worker] | Attempt 25/30 - Server not ready yet... [prefect-worker] | Attempt 26/30 - Server not ready yet... Here is my docker-compose.yaml for reference:
m
thought for 202.1 seconds
c
@Marvin Now in this docker-compose deployment I am getting this error: [prefect-worker] | Attempt 18/30 - Server not ready yet... [prefect-worker] | Attempt 19/30 - Server not ready yet... [prefect-worker] | Attempt 20/30 - Server not ready yet... [prefect-worker] | Attempt 21/30 - Server not ready yet... [prefect-worker] | Attempt 22/30 - Server not ready yet... [prefect-worker] | Attempt 23/30 - Server not ready yet... [prefect-worker] | Attempt 24/30 - Server not ready yet... [prefect-worker] | Attempt 25/30 - Server not ready yet... [prefect-worker] | Attempt 26/30 - Server not ready yet... Here is my docker-compose.yaml for reference:
Copy code
services:
  postgres:
    image: <http://artifactory.company.com/docker-all/postgres:16|artifactory.company.com/docker-all/postgres:16>
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: prefect
      POSTGRES_DB: prefect
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U prefect"]
      interval: 5s
      timeout: 5s
      retries: 5

  redis:
    image: <http://artifactory.company.com/docker-all/redis:7|artifactory.company.com/docker-all/redis:7>
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping"]
      interval: 5s
      timeout: 5s
      retries: 5

  prefect-server:
    image: <http://artifactory.company.com/docker-all/prefecthq/prefect:3-latest|artifactory.company.com/docker-all/prefecthq/prefect:3-latest>
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
      PREFECT_SERVER_API_HOST: 0.0.0.0
      PREFECT_SERVER_API_PORT: 4200
      PREFECT_UI_URL: http://${HOSTNAME:-localhost}:4200
      PREFECT_UI_API_URL: http://${HOSTNAME:-localhost}:4200/api
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: 6379
      PREFECT_REDIS_MESSAGING_DB: 0
    command: sh -c "prefect server database upgrade --yes && prefect server start --no-services"
    ports:
      - "4200:4200"
    healthcheck:
      test: ["CMD-SHELL", "curl -f <http://localhost:4200/api/ready> || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s

  prefect-services:
    image: <http://artifactory.company.com/docker-all/prefecthq/prefect:3-latest|artifactory.company.com/docker-all/prefecthq/prefect:3-latest>
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: 6379
      PREFECT_REDIS_MESSAGING_DB: 0
    command: prefect server services start

  prefect-worker:
    build:
      context: /company/bundles/maestro/linux64/${VERSION}
      dockerfile: /company/prod/fpa/fpa_maestro_client/maestro_client/conf/Dockerfile
    depends_on:
      prefect-server:
        condition: service_healthy
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      # Prefect Configuration - use localhost since we're in host network mode
      PREFECT_API_URL: <http://localhost:4200/api>
      PREFECT_LOGGING_LEVEL: DEBUG
      # Slack service URL - use actual hostname that works from host
      SLACK_SERVICE_URL: <http://ptt-slack:4509/post>
      # Vault credentials from environment variables
      DECRYPT_VAULT: ${DECRYPT_VAULT}
      PROD_APPROLE_TBM_ROLE_ID: ${PROD_APPROLE_TBM_ROLE_ID}
      PROD_APPROLE_TBM_SECRET_ID: ${PROD_APPROLE_TBM_SECRET_ID}
      PROD_APPROLE_ROLE_ID: ${PROD_APPROLE_ROLE_ID}
      PROD_APPROLE_SECRET_ID: ${PROD_APPROLE_SECRET_ID}
      PROD_APPROLE_HCA_ROLE_ID: ${PROD_APPROLE_HCA_ROLE_ID}
      PROD_APPROLE_HCA_SECRET_ID: ${PROD_APPROLE_HCA_SECRET_ID}
    volumes:
      # NFSv3 mount points - specific directories only
      - /remote/finance:/remote/finance:rw
      - /remote/tbm:/remote/tbm:rw
      - /remote/applications:/remote/applications:rw
      # Cache directory for writable storage
      - maestro_cache:/app/src/maestro/cache:rw
      # Mount prefect.yaml from runtime config location
      - /company/prod/fpa/fpa_maestro_client/maestro_client/conf/prefect.yaml:/app/src/maestro/prefect.yaml:ro
    network_mode: host
    restart: unless-stopped

volumes:
  postgres_data:
  redis_data:
  maestro_cache:
m
thought for 201.2 seconds
Thanks — that “Server not ready yet…” loop usually means the worker can’t reach the server API or is pointing at the wrong URL. It looks like your compose file didn’t come through — can you paste it here (redacting any secrets)? In the meantime, here’s a quick checklist and a minimal working example you can compare against. Checklist - Worker API URL: Make sure the worker has
PREFECT_API_URL
set and it includes the
/api
suffix, e.g.
<http://prefect-server:4200/api>
. Missing
/api
will cause readiness checks to fail. - Service hostname: In Docker Compose, use the server service name as the host (e.g.,
prefect-server
), not
localhost
. - Server bind address: Ensure the server listens on all interfaces so other containers can reach it:
prefect server start --host 0.0.0.0
. - Health + dependency: Add a healthcheck on the server using
/api/ready
and have the worker depend on the server being healthy. - DB ready/migrated: Confirm Postgres is ready and migrations have run (auto-migrate on start or a one-time
prefect server database upgrade -y
). Also ensure
pg_trgm
is enabled. Minimal example
Copy code
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: prefect
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: prefect
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 5s
      retries: 10
    volumes:
      # Enables pg_trgm on first init
      - ./postgres-init:/docker-entrypoint-initdb.d

  prefect-server:
    image: prefecthq/prefect:3-latest
    command: ["prefect", "server", "start", "--ui", "--host", "0.0.0.0"]
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
      PREFECT_API_DATABASE_MIGRATE_ON_START: "true"
      # Optional for slower environments / first run
      PREFECT_API_DATABASE_TIMEOUT: "600"
    ports:
      - "4200:4200"
    depends_on:
      postgres:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-fsS", "<http://localhost:4200/api/ready>"]
      interval: 5s
      timeout: 5s
      retries: 20

  prefect-worker:
    image: prefecthq/prefect:3-latest
    command: ["prefect", "worker", "start", "-p", "my-pool"]
    environment:
      # Critical: include the /api suffix
      PREFECT_API_URL: <http://prefect-server:4200/api>
    depends_on:
      prefect-server:
        condition: service_healthy
Quick diagnostics - From the worker container, try:
curl -v <http://prefect-server:4200/api/ready>
- If this fails, it’s either the wrong host/port or the server isn’t listening on 0.0.0.0. - If it 404s, your
PREFECT_API_URL
likely lacks the
/api
suffix. - If it times out, check server healthcheck logs and Postgres readiness/migrations. Relevant docs - Docker Compose guide: Prefect 3 - Docker Compose - Server CLI (including database upgrade): Server CLI - Settings reference: Settings reference - Server health endpoints: /api/ready and /api/health Please paste your
docker-compose.yml
and I’ll point out the exact changes needed.
Thanks for sharing the compose file — the worker’s “Server not ready yet…” loop usually means it cannot reach the API at the URL you’ve configured. In your case, there are a few likely culprits: 1) The worker is using host networking + “localhost” - With
network_mode: host
,
localhost
inside the worker resolves to the host’s loopback. If your system prefers IPv6 for
localhost
(i.e.,
::1
), it can fail to reach the API if the port is only bound on IPv4. - Quick fix: change the worker’s
PREFECT_API_URL
to use IPv4 explicitly:
Copy code
PREFECT_API_URL: <http://127.0.0.1:4200/api>
Alternatively, remove
network_mode: host
, keep everything on the default Docker network, and set:
Copy code
PREFECT_API_URL: <http://prefect-server:4200/api>
2) Angle brackets around URLs/DSNs in env - In the snippet, I see
<...>
around URLs, e.g.:
Copy code
PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
  PREFECT_API_URL: <http://localhost:4200/api>
If those angle brackets are literally in your compose file, Prefect will get invalid URLs and will never connect. Make sure the env values are plain strings without `<...>`:
Copy code
PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
  PREFECT_API_URL: <http://127.0.0.1:4200/api>
3) UI URL envs - Not directly related to the worker, but these affect the UI:
Copy code
PREFECT_UI_URL: http://${HOSTNAME:-localhost}:4200
  PREFECT_UI_API_URL: http://${HOSTNAME:-localhost}:4200/api
Inside the container,
HOSTNAME
is the container ID and won’t be reachable from your browser. Use a hostname your browser can reach (e.g., the machine’s DNS name or IP). For local testing:
Copy code
PREFECT_UI_URL: <http://localhost:4200>
  PREFECT_UI_API_URL: <http://localhost:4200/api>
4) Verify server readiness from the worker container - This will immediately tell us if it’s a reachability/URL issue:
Copy code
docker compose exec prefect-worker sh -lc 'curl -v <http://127.0.0.1:4200/api/ready>'
If that fails, try from the host itself:
Copy code
curl -v <http://127.0.0.1:4200/api/ready>
You should see a 200 when the API is ready. Docs on the endpoint: Perform Readiness Check Minimal changes I recommend right now - Remove angle brackets from URLs/DSNs - Pin the worker API URL to IPv4 or move it to the same Docker network as the server Example adjusted snippet:
Copy code
prefect-server:
    ...
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
      PREFECT_SERVER_API_HOST: 0.0.0.0
      PREFECT_SERVER_API_PORT: 4200
      PREFECT_UI_URL: <http://localhost:4200>
      PREFECT_UI_API_URL: <http://localhost:4200/api>
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: 6379
      PREFECT_REDIS_MESSAGING_DB: 0

  prefect-worker:
    ...
    # Option A: keep host networking and force IPv4 loopback
    network_mode: host
    environment:
      PREFECT_API_URL: <http://127.0.0.1:4200/api>
      PREFECT_LOGGING_LEVEL: DEBUG

    # Option B: recommended — remove host networking and use service DNS
    # (comment out network_mode)
    # environment:
    #   PREFECT_API_URL: <http://prefect-server:4200/api>
If this doesn’t resolve it, please: - Confirm whether the
<...>
brackets are actually present in your compose file - Share the output of: -
docker compose logs -f prefect-server
-
docker compose exec prefect-worker sh -lc 'env | grep PREFECT_API_URL && curl -sv <http://127.0.0.1:4200/api/ready>'
Relevant docs: - Docker Compose guide: Prefect 3 - Docker Compose
- Server readiness endpoint: Perform Readiness Check