https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I am deploying via docker-compose. My...
# ask-marvin
c
@Marvin I am deploying via docker-compose. My logs show that the worker has been created, deployment has been done successfully, and that it is using my host URL (which is defined in docker compose), but when I go to this URL, nothing shows up in /work-pools or /deployments
m
✅ thought for 292.4 seconds
This symptom almost always means the UI is talking to a different Prefect API than the one your worker/CLI used to create the work pool and deployment. Quick checks to confirm and fix: 1) Verify the API your browser UI is calling - Open the UI, press F12 → Network → filter for “api”. You should see requests to something like http//&lt;your host&gt;4200/api/… - If those calls are going to a different host (or failing), the UI is pointed wrong. In a standard docker-compose setup, the UI should call the same origin (no special env needed). If you set PREFECT_UI_API_URL, make sure it points to the externally reachable address, e.g. http://localhost:4200/api Docs: Self-hosted with docker-compose and Settings and profiles 2) Confirm the server you intend to use is reachable - From your host, hit the health endpoint:
Copy code
curl -s http://<your-host-or-ip>:4200/api/health
You should get JSON back indicating server readiness. 3) Point your CLI at that exact server and list objects - From the same environment where you ran prefect deploy:
Copy code
PREFECT_API_URL=http://<your-host-or-ip>:4200/api prefect work-pool ls
PREFECT_API_URL=http://<your-host-or-ip>:4200/api prefect deployment ls
- If these are empty, then your deployment and pool were created against a different API. Run:
Copy code
prefect config view
prefect profile inspect
to see which PREFECT_API_URL was active when you ran the deploy. It’s common for the CLI to be pointed at one API (e.g., a different docker-compose stack or a local server you started earlier) while the UI is pointed at another. 4) Check environment mismatches (common on Linux) - Inside containers (including a worker running in compose), use service names like http://api:4200/api to reach the API. - From your host browser and host CLI, use the host-mapped address: http://localhost:4200/api or http//&lt;server hostname&gt;4200/api - host.docker.internal does not work on Linux by default; avoid using it on RHEL9. 5) Ensure DB persistence wasn’t lost - If your Postgres volume is ephemeral or got reset, prior pools/deployments would disappear on restart. - Confirm your compose has a persistent volume for Postgres and that migrations completed:
Copy code
docker compose ps
docker compose logs -f
Look for messages that the API is serving and DB migrations are done. Reference env you generally want in compose (abridged):
Copy code
services:
  api:
    image: prefecthq/prefect:3-latest
    command: prefect server start
    ports:
      - "4200:4200"
    environment:
      PREFECT_SERVER_API_HOST: "0.0.0.0"
      PREFECT_API_DATABASE_CONNECTION_URL: "<postgresql+asyncpg://postgres:postgres@postgres:5432/prefect>"
      # Optional: only if serving UI from a different origin
      # PREFECT_UI_API_URL: "<http://localhost:4200/api>"

  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: prefect
    volumes:
      - prefect_pg:/var/lib/postgresql/data

volumes:
  prefect_pg:
Given your setup (RHEL9, process worker, need host filesystem access), running the worker on the host is fine; just make sure your host worker and your deploy CLI both use the same PREFECT_API_URL as the UI you’re visiting. If you can, please share: - The exact URL you’re opening in the browser - Output of:
Copy code
prefect version
prefect config view
prefect profile inspect
PREFECT_API_URL=http://<host>:4200/api prefect work-pool ls
PREFECT_API_URL=http://<host>:4200/api prefect deployment ls
curl -s http://<host>:4200/api/health
- The relevant env section from your docker-compose.yml (redact secrets) With those, I can pinpoint whether it’s a UI API target, CLI profile, or DB persistence issue.
c
@Marvin in the network tab after filtering for api I see this: api-D0hrn_eh.js 304 script index.ts:42 0.2 kB 100 ms health (failed)net::ERR_CONNECTION_REFUSED xhr AppRouterView.vue:62 0.0 kB 2.37 s filter (failed)net::ERR_CONNECTION_REFUSED xhr WorkPools.vue:30 0.0 kB 2.40 s health (failed)net::ERR_CONNECTION_REFUSED preflight Preflight 0.0 kB 2.31 s filter (failed)net::ERR_CONNECTION_REFUSED preflight Preflight 0.0 kB 2.30 s
m
✅ thought for 45.4 seconds
Thanks — those Network tab errors confirm the UI can’t reach the Prefect API (connection refused). That’s almost always an API URL/port mismatch. Quick checks to fix: 1) See exactly where the UI is trying to call - In DevTools, click one failed request and copy the Request URL (full host:port/path). Please share it here. - The correct pattern should be http//&lt;your host or ip&gt;4200/api/health (and /api/work-pools/filter, etc.) - Two common misconfigs: - PREFECT_UI_API_URL is set to an internal service name (e.g., http://api:4200/api). That works inside containers but fails in your browser on the host. - PREFECT_UI_API_URL is missing the /api suffix (so requests go to /health instead of /api/health). 2) Test that exact URL from your host
Copy code
curl -v http://<host-or-ip>:4200/api/health
If this fails, the API isn’t reachable from your host (port not exposed, server not listening on 0.0.0.0, firewall, or reverse proxy issue). 3) Check docker-compose for port and env - Ensure the API service publishes port 4200:
Copy code
ports:
  - "4200:4200"
- Ensure the server binds to all interfaces in the container:
Copy code
environment:
  PREFECT_SERVER_API_HOST: "0.0.0.0"
- Unless you’re reverse-proxying, either: - Do not set PREFECT_UI_API_URL (the UI will call same-origin /api), or - Set it to an external address reachable by your browser, including /api: PREFECT_UI_API_URL: "http//&lt;host or ip&gt;4200/api" - Avoid setting PREFECT_UI_API_URL to a container-only hostname like api or prefect-server. 4) Verify the server is healthy
Copy code
docker compose logs -f
Look for API startup and DB migrations completed. If Postgres was reset (no persistent volume), any prior pools/deployments would be gone. 5) Sanity-check with the CLI against the same URL your browser should use
Copy code
PREFECT_API_URL=http://<host-or-ip>:4200/api prefect work-pool ls
PREFECT_API_URL=http://<host-or-ip>:4200/api prefect deployment ls
If you can, please share: - The Request URL from one failed request in DevTools - The output of:
Copy code
curl -v http://<host-or-ip>:4200/api/health
- The relevant part of your docker-compose.yml for the Prefect server service (ports + environment) (redact secrets) With that, I can point to the exact setting to change.
c
@Marvin here is an example of a failed URL: http://localhost:4200/api/flow_runs/filter . Here is my docker-compose for reference: services: postgres: image: artifactory.chicagotrading.com/docker-all/postgres:16 environment: POSTGRES_USER: prefect POSTGRES_PASSWORD: prefect POSTGRES_DB: prefect volumes: - postgres_data:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -U prefect"] interval: 5s timeout: 5s retries: 5 redis: image: artifactory.chicagotrading.com/docker-all/redis:7 volumes: - redis_data:/data healthcheck: test: ["CMD-SHELL", "redis-cli ping"] interval: 5s timeout: 5s retries: 5 prefect-server: image: artifactory.chicagotrading.com/docker-all/prefecthq/prefect:3-latest depends_on: postgres: condition: service_healthy redis: condition: service_healthy environment: PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:prefect@postgres:5432/prefect PREFECT_SERVER_API_HOST: 0.0.0.0 PREFECT_SERVER_API_PORT: 4200 PREFECT_UI_URL: http://${HOSTNAME localhost}4200 PREFECT_UI_API_URL: http://${HOSTNAME localhost}4200/api PREFECT_MESSAGING_BROKER: prefect_redis.messaging PREFECT_MESSAGING_CACHE: prefect_redis.messaging PREFECT_REDIS_MESSAGING_HOST: redis PREFECT_REDIS_MESSAGING_PORT: 6379 PREFECT_REDIS_MESSAGING_DB: 0 command: sh -c "echo '[SERVER] Starting database upgrade...' && prefect server database upgrade --yes && echo '[SERVER] Database upgrade complete, starting server...' && prefect server start --no-services" ports: - "4200:4200" healthcheck: test: ["CMD-SHELL", "echo '[HEALTH] Testing API readiness...' && curl -f -v http://localhost:4200/api/ready 2>&1 | tee /dev/stderr || (echo '[HEALTH] API not ready yet' && exit 1)"] interval: 10s timeout: 5s retries: 5 start_period: 30s prefect-services: image: artifactory.chicagotrading.com/docker-all/prefecthq/prefect:3-latest depends_on: postgres: condition: service_healthy redis: condition: service_healthy environment: PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:prefect@postgres:5432/prefect PREFECT_MESSAGING_BROKER: prefect_redis.messaging PREFECT_MESSAGING_CACHE: prefect_redis.messaging PREFECT_REDIS_MESSAGING_HOST: redis PREFECT_REDIS_MESSAGING_PORT: 6379 PREFECT_REDIS_MESSAGING_DB: 0 command: sh -c "echo '[SERVICES] Starting Prefect services...' && prefect server services start" prefect-worker: build: context: /ctc dockerfile: prod/fpa/fpa_maestro_client/maestro_client/conf/Dockerfile args: VERSION: ${VERSION} depends_on: prefect-server: condition: service_healthy postgres: condition: service_healthy redis: condition: service_healthy environment: # Bundle version for environment detection VERSION: ${VERSION} # Prefect Configuration - use IPv4 localhost to avoid IPv6 resolution issues PREFECT_API_URL: http://127.0.0.1:4200/api PREFECT_LOGGING_LEVEL: DEBUG # Slack service URL - use actual hostname that works from host SLACK_SERVICE_URL: http://ptt-slack:4509/post # Vault credentials from environment variables DECRYPT_VAULT: ${DECRYPT_VAULT} PROD_APPROLE_TBM_ROLE_ID: ${PROD_APPROLE_TBM_ROLE_ID} PROD_APPROLE_TBM_SECRET_ID: ${PROD_APPROLE_TBM_SECRET_ID} PROD_APPROLE_ROLE_ID: ${PROD_APPROLE_ROLE_ID} PROD_APPROLE_SECRET_ID: ${PROD_APPROLE_SECRET_ID} PROD_APPROLE_HCA_ROLE_ID: ${PROD_APPROLE_HCA_ROLE_ID} PROD_APPROLE_HCA_SECRET_ID: ${PROD_APPROLE_HCA_SECRET_ID} volumes: # NFSv3 mount points - specific directories only - /remote/finance/remote/financerw - /remote/tbm/remote/tbmrw - /remote/applications/remote/applicationsrw # Cache directory for writable storage - maestro_cache/app/src/maestro/cacherw # prefect.yaml now copied directly in Dockerfile (no volume mount needed) network_mode: host restart: unless-stopped volumes: postgres_data: redis_data: maestro_cache:
m
✅ thought for 54.1 seconds
Thanks for the details — the failed Request URL confirms it: the UI is trying to call http://localhost:4200/api/…, but from your browser’s perspective “localhost” is your own machine, not the RHEL9 server. That’s why you’re getting ERR_CONNECTION_REFUSED. Root cause - You’ve set PREFECT_UI_API_URL to http://${HOSTNAME localhost}4200/api inside the prefect-server container. In the browser, the UI uses that literal value. If you open the UI from another machine, “localhost:4200” points to your laptop, not the server. Two ways to fix Option A (recommended): Don’t set PREFECT_UI_API_URL at all - The UI will call the same origin that served it, i.e., if you visit http://your-server:4200 the UI will call http://your-server:4200/api automatically. - Remove PREFECT_UI_URL and PREFECT_UI_API_URL from the server service. Option B: Set PREFECT_UI_API_URL to your server’s actual hostname - If you must hardcode it, set it to the externally reachable host and include /api, e.g.: PREFECT_UI_API_URL: "http://your-server.domain:4200/api" Suggested docker-compose changes (abridged)
Copy code
services:
  prefect-server:
    image: <http://artifactory.chicagotrading.com/docker-all/prefecthq/prefect:3-latest|artifactory.chicagotrading.com/docker-all/prefecthq/prefect:3-latest>
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: "<postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>"
      PREFECT_SERVER_API_HOST: "0.0.0.0"
      PREFECT_SERVER_API_PORT: "4200"
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: 6379
      PREFECT_REDIS_MESSAGING_DB: 0
      # Remove these (let UI use same-origin), OR set to your actual host if needed
      # PREFECT_UI_URL: http://${HOSTNAME:-localhost}:4200
      # PREFECT_UI_API_URL: http://${HOSTNAME:-localhost}:4200/api
    ports:
      - "4200:4200"
    command: >
      sh -c "prefect server database upgrade --yes &&
             prefect server start --no-services"
After changing: 1) Restart the server and hard refresh the browser (Ctrl+Shift+R). 2) In DevTools → Network, the failed Request URL should now be http://your-server:4200/api/… (not localhost). 3) From your workstation, verify:
Copy code
curl -v <http://your-server:4200/api/health>
Notes about your worker - You’ve set network_mode: host and PREFECT_API_URL: http://127.0.0.1:4200/api — that’s valid because the worker runs on the same host that exposes port 4200. - If you ever run the worker from another host/container network, change PREFECT_API_URL to the server’s hostname. If you still don’t see pools/deployments after fixing the UI API URL, we should confirm they exist on this server (not another one) by running from your workstation:
Copy code
PREFECT_API_URL=<http://your-server:4200/api> prefect work-pool ls
PREFECT_API_URL=<http://your-server:4200/api> prefect deployment ls
Docs for reference: - Self-hosted with docker-compose - Settings and profiles If you share the exact UI URL you’re opening and the result of curl -v http://your-server:4200/api/health from your workstation, I can confirm everything’s wired correctly.
c
I did this fix but now getting this network response: Request URL http://0.0.0.0:4200/api/health Referrer Policy strict-origin-when-cross-origin accept application/json, text/plain, / referer http://ch12lxtrd19:4200/ user-agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36 x-prefect-ui true
@Marvin I did this fix but now getting this network response: Request URL http://0.0.0.0:4200/api/health Referrer Policy strict-origin-when-cross-origin accept application/json, text/plain, / referer http://ch12lxtrd19:4200/ user-agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36 x-prefect-ui true
m
✅ thought for 57.3 seconds
Got it — the UI is now calling http://0.0.0.0:4200/api/health. 0.0.0.0 is a bind address, not a routable host, so the browser will always fail. This means the running server has PREFECT_UI_API_URL set (directly or via templating) to http://0.0.0.0:4200/api. Fix - Best: unset PREFECT_UI_API_URL (and PREFECT_UI_URL). The UI will use same-origin, so if you open http://ch12lxtrd19:4200, it will call http://ch12lxtrd19:4200/api automatically. - If you must set it, use your actual host: PREFECT_UI_API_URL: "http://ch12lxtrd19:4200/api" or use a relative path: PREFECT_UI_API_URL: "/api" Make sure the container picks up the change 1) Update docker-compose.yml (remove PREFECT_UI_URL and PREFECT_UI_API_URL or set correctly). 2) Recreate the server container:
Copy code
docker compose up -d --force-recreate --no-deps prefect-server
3) Hard refresh the browser (Ctrl+Shift+R) to bust cached UI config. Verify what the running container has
Copy code
docker compose exec prefect-server sh -lc 'env | sort | grep PREFECT_UI'
If you still see PREFECT_UI_API_URL=http://0.0.0.0:4200/api, it didn’t pick up the change (or is coming from a .env). Remove it and recreate again. Sanity checks - From your workstation:
Copy code
curl -v <http://ch12lxtrd19:4200/api/health>
- In the browser Network tab, confirm the Request URL is now http://ch12lxtrd19:4200/api/... (not 0.0.0.0, not localhost). If after this the UI loads but looks empty, confirm objects exist on this server:
Copy code
PREFECT_API_URL=<http://ch12lxtrd19:4200/api> prefect work-pool ls
PREFECT_API_URL=<http://ch12lxtrd19:4200/api> prefect deployment ls
If anything’s still off, please share: - Output of the env check inside the prefect-server container - The updated prefect-server environment section from docker-compose - The new Request URL from a failed/working call in DevTools
c
@Marvin I already removed REFECT_UI_API_URL (and PREFECT_UI_URL) and get this same error. At one point I am running prefect config set PREFECT_API_URL=http://ch12lxtrd19:4200/api . Could this be causing the issue?
m
✅ thought for 266.1 seconds
Yes — that explains it. What’s happening - When PREFECT_UI_API_URL is unset, the server generates the UI’s API URL from its own settings. Because your server is binding to 0.0.0.0, the UI config ends up as http://0.0.0.0:4200/api — which is not a routable host in a browser. - Running prefect config set PREFECT_API_URL=... on your workstation does not affect the server container. That command only updates your local Prefect profile; the server still derived 0.0.0.0. Fix Set the UI API URL inside the prefect-server container to a browser-reachable address (or to a same-origin relative path): Option A (explicit host) - In prefect-server’s environment: PREFECT_UI_API_URL: "http://ch12lxtrd19:4200/api" Option B (same-origin) - In prefect-server’s environment: PREFECT_UI_API_URL: "/api" Then recreate the server container and hard refresh the UI:
Copy code
docker compose up -d --force-recreate --no-deps prefect-server
# verify what the container is actually using
docker compose exec prefect-server sh -lc 'env | sort | grep PREFECT_UI'
# hard refresh the browser (Ctrl+Shift+R)
Sanity checks - From your workstation:
Copy code
curl -v <http://ch12lxtrd19:4200/api/health>
- In the browser Network tab, confirm Request URL is now http://ch12lxtrd19:4200/api/... (not 0.0.0.0). Notes - Keep PREFECT_SERVER_API_HOST=0.0.0.0 so the API binds to all interfaces in the container; just don’t let that leak into the UI config. - Your previous prefect config set PREFECT_API_URL=... on your workstation is fine for your CLI/worker, but it won’t configure the server UI. Use the env var in docker-compose for that. Docs - Self-hosted with docker-compose: Guide - Settings and profiles: Reference If you still see 0.0.0.0 in the request URL after the change, please share the output of:
Copy code
docker compose exec prefect-server sh -lc 'env | sort | grep -E "PREFECT_UI|PREFECT_API_URL|PREFECT_SERVER_API_HOST"'
and the current prefect-server environment section from docker-compose, and I’ll pinpoint what’s still injecting 0.0.0.0.
36 Views
Open in Slack
PreviousNext
Powered by