Kurt Sys (Vintecc)
10/07/2025, 3:55 PMFile "/usr/local/lib/python3.10/socket.py", line 967, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -5] No address associated with hostname An exception occurred.
So, although I have a perfectly fine running work pool and worker, when triggering a run, it seems the address/hostname is wrongMarvin
10/07/2025, 3:55 PMMarvin
10/07/2025, 4:02 PMPREFECT_API_URL. In Docker, localhost points to the container itself, and service names only resolve on the same user-defined network. Your worker can reach the API (polling works) but the flow-run container (spawned by the docker worker) likely isn’t on the same network or is given an API URL that isn’t resolvable.
Fix options (pick one):
1) Use your Compose network + service name (recommended)
- Put your Prefect Server and Worker services on a named user-defined network (e.g., prefect-net).
- Make sure the docker worker launches flow-run containers on that same network via the work pool’s base job template.
- Set PREFECT_API_URL to the server’s service name on that network (e.g., `http://prefect-server:4200/api`).
Example docker-compose.yml (abridged):
networks:
prefect-net:
services:
prefect-server:
image: prefecthq/prefect:3-latest
command: prefect server start
networks: [prefect-net]
# expose/mapping: "4200:4200" for host access
prefect-worker:
image: prefecthq/prefect:3-latest
command: prefect worker start -p docker-pool
environment:
PREFECT_API_URL: <http://prefect-server:4200/api>
networks: [prefect-net]
Create/update your docker work pool so flow-run containers join that same network:
prefect work-pool get-default-base-job-template --type docker --file base-template.json
# Edit to include at least:
# {
# "networks": ["prefect-net"],
# "env": {"PREFECT_API_URL": "<http://prefect-server:4200/api>"}
# }
prefect work-pool create docker-pool --type docker --base-job-template base-template.json --overwrite
2) Use the host gateway (Linux) with host.docker.internal
- If your server is on the host, set PREFECT_API_URL=<http://host.docker.internal:4200/api>.
- Ensure flow-run containers can resolve host.docker.internal by adding an extra host mapping via the work pool template:
{
"env": {
"PREFECT_API_URL": "<http://host.docker.internal:4200/api>"
},
"container_create_kwargs": {
"extra_hosts": {
"host.docker.internal": "host-gateway"
}
}
}
3) Use your host’s LAN IP
- Set PREFECT_API_URL to http://<host-ip>:4200/api. Ensure networking allows access from containers.
Why this happens
- The worker passes its Prefect settings (including PREFECT_API_URL) into the flow-run container. If that hostname isn’t resolvable from the flow-run container’s network context, you’ll see socket.gaierror: [Errno -5] No address associated with hostname.
Quick checks
- In the worker and flow-run containers, run:
prefect config view | grep PREFECT_API_URL
- From a temporary container on the intended network, verify DNS:
docker run --rm --network prefect-net alpine sh -c "apk add --no-cache bind-tools >/dev/null && nslookup prefect-server"
If you’d like, share:
- Your docker-compose.yml (services + networks section)
- The work pool name and its base job template (prefect work-pool inspect <pool>)
- What PREFECT_API_URL resolves to in your worker container
Related docs
- Self-hosted Prefect Server with Docker
- Manage settings (PREFECT_API_URL)
- Blog: Work pools and workers overviewKurt Sys (Vintecc)
10/07/2025, 4:07 PMnetworks in every container, so they can find each other and I have the hard coded value? I can add multiple networks for each service, and it will look in all these networks for the right container?
btw, I'm using prefect 3.x?Marvin
10/07/2025, 4:10 PMKurt Sys (Vintecc)
10/07/2025, 4:10 PMdeployment_id = trigger_flow.deploy(
name="fraio-trigger",
work_pool_name="fraio",
image="harbor.mgmt.vintecc.cloud/vintecc-ags-hbm/fraio_algo:local",
build=False,
push=False,
version="local",
tags=[],
description=None,
entrypoint_type=EntrypointType.MODULE_PATH,
concurrency_limit=5,
print_next_steps=True,
infrastructure_overrides={
"env": {
"PREFECT_API_URL": "<http://prefect:4200/api>",
},
"network": "agristoags-hbm_default", # ensure same network
},
)Marvin
10/07/2025, 4:10 PMMarvin
10/07/2025, 4:10 PMprefect-server). Using the service name in PREFECT_API_URL is the right approach.
- Can I attach multiple networks and it will “look in all of them”? Not exactly. Docker’s DNS will only resolve names for containers that share a network with the caller. If two containers share multiple networks, the name still resolves (to the peer’s IP on one of the shared networks). Docker does not “bridge” networks for you. Practically:
- Ensure the flow-run container and the Prefect Server share at least one network.
- To avoid ambiguity and surprises, prefer a single shared network for all Prefect services and flow-run containers (e.g., prefect-net).
Important Prefect-specific note
- Your flow runs are launched by the Docker worker as new containers outside docker-compose. They won’t be on your compose network unless you tell the work pool to attach them. Set networks in the Docker work pool’s base job template so every flow-run container joins the same network as your server.
Example base job template for the Docker work pool
{
"image": "your/image:tag",
"env": {
"PREFECT_API_URL": "<http://prefect-server:4200/api>"
},
"networks": [
"prefect-net"
]
}
How to apply the base job template
prefect work-pool get-default-base-job-template --type docker --file base-template.json
# edit base-template.json as above
prefect work-pool create docker-pool --type docker --base-job-template base-template.json --overwrite
Example docker-compose sketch
networks:
prefect-net:
services:
prefect-server:
image: prefecthq/prefect:3-latest
command: prefect server start
networks: [prefect-net]
ports:
- "4200:4200"
prefect-worker:
image: prefecthq/prefect:3-latest
command: prefect worker start -p docker-pool
environment:
PREFECT_API_URL: <http://prefect-server:4200/api>
networks: [prefect-net]
Alternative if your server runs on the host (not in Docker) on Linux
- Use the host gateway mapping so containers can reach the host:
{
"env": {
"PREFECT_API_URL": "<http://host.docker.internal:4200/api>"
},
"container_create_kwargs": {
"extra_hosts": {
"host.docker.internal": "host-gateway"
}
}
}
Quick sanity checks
- In both your worker container and any flow-run container:
prefect config view | grep PREFECT_API_URL
- DNS on the network:
docker run --rm --network prefect-net alpine sh -c "apk add --no-cache bind-tools >/dev/null && nslookup prefect-server"
Docs and references
- Self-hosted Prefect Server with Docker
- Manage settings (PREFECT_API_URL)
If you want, share your compose file (services + networks) and the work pool’s base job template (prefect work-pool inspect <pool>), and I’ll help you tighten it up.Marvin
10/07/2025, 4:12 PMjob_variables (not infrastructure_overrides)
- For Docker, the field is networks (list), not network
- Make sure PREFECT_API_URL is a plain URL string without angle brackets
Corrected example:
deployment_id = trigger_flow.deploy(
name="fraio-trigger",
work_pool_name="fraio", # must be a Docker work pool
image="harbor.mgmt.vintecc.cloud/vintecc-ags-hbm/fraio_algo:local",
build=False,
push=False,
version="local",
entrypoint_type=EntrypointType.MODULE_PATH,
concurrency_limit=5,
print_next_steps=True,
job_variables={
"env": {
"PREFECT_API_URL": "<http://prefect:4200/api>"
},
"networks": ["agristoags-hbm_default"], # ensure same network as your server service
# optional: if your server is on the host instead of the same network:
# "container_create_kwargs": {
# "extra_hosts": {"host.docker.internal": "host-gateway"}
# }
},
)
Why this matters:
- The docker worker creates a new container for each flow run. That container must be able to resolve the Prefect API hostname. Using the same user-defined network and a service name (e.g., prefect) is the most reliable approach.
- socket.gaierror: [Errno -5] often appears when the hostname in PREFECT_API_URL is not resolvable from the flow-run container. Angle brackets in the URL (e.g., "<http://prefect:4200/api>") will also break DNS resolution — use <http://prefect:4200/api>.
Quick verifications:
- Confirm your work pool is Docker: prefect work-pool inspect fraio
- See allowed job variables: prefect work-pool get-default-base-job-template --type docker
- In a running flow-run container, check settings: prefect config view | grep PREFECT_API_URL
If you still hit issues, please share:
- The prefect work-pool inspect fraio output
- The service and network names in your docker-compose.yml
References:
- Customize job variables
- Manage settings (PREFECT_API_URL)
- Self-hosted Prefect Server with DockerKurt Sys (Vintecc)
10/07/2025, 4:23 PMnetworks:
prefect-net:
services:
prefect:
build:
context: .
dockerfile: Dockerfile.prefect
container_name: prefect
restart: always
volumes:
- prefect:/root/.prefect
entrypoint: ["/opt/prefect/entrypoint.sh", "prefect", "server", "start"]
environment:
- PREFECT_UI_URL=<http://127.0.0.1:4200/ui>
- PREFECT_SERVER_API_HOST=0.0.0.0
- PREFECT_API_DATABASE_CONNECTION_URL=postgresql+asyncpg://${ORCHESTRATOR_DB_USER}:${ORCHESTRATOR_DB_PASSWORD}@${ORCHESTRATOR_DB_HOST}:${ORCHESTRATOR_DB_PORT}/${ORCHESTRATOR_DB_NAME}
ports:
- 4200:4200
depends_on:
- fraio-data
healthcheck:
test: ["CMD", "curl", "-fsS", "<http://localhost:4200/api/health>"]
interval: 5s
timeout: 3s
retries: 20
networks: [prefect-net]
profiles:
- dev
prefect-workpool:
build:
context: .
dockerfile: Dockerfile.prefect-workpool
depends_on:
prefect:
condition: service_healthy
environment:
- PREFECT_API_URL=<http://prefect:4200/api>
volumes:
- /var/run/docker.sock:/var/run/docker.sock
entrypoint: ["/bin/bash", "-c"]
command: |
"
set -e
echo 'Creating work pool...'
prefect work-pool create fraio -t docker --overwrite
echo 'Starting worker for fraio...'
prefect worker start --pool 'fraio'
"
networks: [prefect-net]
profiles:
- devMarvin
10/07/2025, 4:23 PMKurt Sys (Vintecc)
10/07/2025, 4:27 PMroot@9bf7f4cf741d:/opt/prefect# prefect config view | grep PREFECT_API_URL
root@9bf7f4cf741d:/opt/prefect# prefect config view
🚀 you are connected to:
<http://127.0.0.1:4200/ui>
PREFECT_PROFILE='ephemeral'
PREFECT_API_DATABASE_CONNECTION_URL='********' (from env)
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='true' (from profile)
PREFECT_SERVER_API_HOST='0.0.0.0' (from env)
PREFECT_UI_URL='<http://127.0.0.1:4200/ui>' (from env)Marvin
10/07/2025, 4:27 PMMarvin
10/07/2025, 4:28 PMPREFECT_API_DATABASE_CONNECTION_URL=postgresql+asyncpg://... is supported in Prefect 3.x. Prefect will auto-create the schema on first start.
- Exposing the server on 0.0.0.0 is supported. PREFECT_SERVER_API_HOST maps to the --host flag for prefect server start (confirmed by prefect server start --help).
- PREFECT_UI_URL is the right setting to advertise an externally reachable UI URL.
- Docker worker setup:
- prefect work-pool create fraio -t docker --overwrite is valid.
- prefect worker start --pool fraio is valid.
- Mounting /var/run/docker.sock is required for the Docker worker.
- Pointing the worker at the API with PREFECT_API_URL=<http://prefect:4200/api> (using the compose service name) is correct.
- Healthcheck against <http://localhost:4200/api/health> is appropriate.
Things to fix or double-check
- Remove Slack-style angle brackets from URLs in env vars. In compose they’ll be literal. For example:
- PREFECT_UI_URL=<http://127.0.0.1:4200/ui>
- PREFECT_API_URL=<http://prefect:4200/api>
- Ensure the image you build actually has /opt/prefect/entrypoint.sh. That script exists in the official Prefect images; if you’re building your own, either copy it or switch to a command: ["prefect","server","start"] style.
- Your worker command is wrapped in extra quotes. Those quotes will be passed to bash and can break the script. Remove the outer quotes.
- Make sure curl is present in the server image if you keep the curl-based healthcheck.
- You’re using a named volume prefect:/root/.prefect but didn’t declare it at the top-level. Add a volumes: section.
- If your DB is a separate compose service, add a proper depends_on (with a healthcheck) to ensure Postgres is ready before the API starts.
A tightened compose example
yaml
networks:
prefect-net:
volumes:
prefect:
services:
prefect:
# If you prefer, you can use the official image instead of build:
# image: prefecthq/prefect:3-python3.10
build:
context: .
dockerfile: Dockerfile.prefect
container_name: prefect
restart: unless-stopped
volumes:
- prefect:/root/.prefect
# If your image includes the official entrypoint script, keep this.
# Otherwise, prefer the command below.
# entrypoint: ["/opt/prefect/entrypoint.sh", "prefect", "server", "start"]
command: ["prefect", "server", "start", "--host", "0.0.0.0", "--port", "4200"]
environment:
- PREFECT_UI_URL=<http://127.0.0.1:4200/ui>
- PREFECT_SERVER_API_HOST=0.0.0.0
- PREFECT_API_DATABASE_CONNECTION_URL=postgresql+asyncpg://${ORCHESTRATOR_DB_USER}:${ORCHESTRATOR_DB_PASSWORD}@${ORCHESTRATOR_DB_HOST}:${ORCHESTRATOR_DB_PORT}/${ORCHESTRATOR_DB_NAME}
ports:
- "4200:4200"
# If you have a Postgres service, add:
# depends_on:
# postgres:
# condition: service_healthy
healthcheck:
test: ["CMD", "curl", "-fsS", "<http://localhost:4200/api/health>"]
interval: 5s
timeout: 3s
retries: 20
networks: [prefect-net]
profiles: [dev]
prefect-workpool:
# image: prefecthq/prefect:3-python3.10
build:
context: .
dockerfile: Dockerfile.prefect-workpool
depends_on:
prefect:
condition: service_healthy
environment:
- PREFECT_API_URL=<http://prefect:4200/api>
volumes:
- /var/run/docker.sock:/var/run/docker.sock
restart: unless-stopped
entrypoint: ["/bin/bash", "-lc"]
command: |
set -e
echo 'Creating work pool...'
prefect work-pool create fraio --type docker --overwrite
echo 'Starting worker for fraio...'
prefect worker start --pool fraio
networks: [prefect-net]
profiles: [dev]
NotesMarvin
10/07/2025, 4:28 PMPREFECT_UI_URL to that public URL.
- Switching from SQLite to Postgres creates a new, empty API DB; Prefect will migrate/create tables automatically.
- You can optionally set a worker name with --name for easier identification.
Docs
- Server via Docker: Self-hosted Prefect Server (Docker)
- Settings reference: Settings reference
- Work pools and workers overview: Work pools and workers
Questions for you
- Are you basing your images on the official Prefect image? If not, do you want to switch to command: ["prefect", "server", "start", ...] instead of relying on /opt/prefect/entrypoint.sh?
- Do you want a sample Postgres service and healthcheck wired into depends_on?Marvin
10/07/2025, 4:32 PMPREFECT_API_URL in your server container:
- PREFECT_API_URL (client-side): This tells Prefect clients (CLI, workers, flow code) where to reach the API. It should point at your server’s externally reachable API endpoint, typically http://<host>:4200/api. You set this on machines running workers/CLI/flows — not in the server container — so it’s normal that prefect config view inside the server does not show it unless you’ve explicitly set it there.
- `PREFECT_SERVER_API_HOST`/`PORT`/`BASE_PATH` (server-side): Controls how the API binds inside the server. Defaults are host 127.0.0.1, port 4200, and base path unset. You’ve set PREFECT_SERVER_API_HOST=0.0.0.0, which is correct to listen on all interfaces. Make sure you expose/map port 4200 from your container/host so clients can reach it.
- PREFECT_UI_URL (client convenience): Used by CLI/clients to print a clickable UI link. It doesn’t affect API connectivity. Set this on your workers/CLI to your externally reachable UI URL (e.g., http://<host>:4200/ui) so “View in UI” links are correct.
- UI → API connection: The server’s UI will call the API at the same origin by default (i.e., it assumes /api on the same host/port it’s served from). You can override it with PREFECT_SERVER_UI_API_URL if you’re fronting the UI/API with a proxy or using nonstandard paths. Otherwise you usually don’t need to set this.
What to set so external workers/CLI can connect
- On each worker/CLI machine (or in your worker container/pod environment), set:
export PREFECT_API_URL=http://<your-server-host>:4200/api
Or persist it in the active profile:
prefect config set PREFECT_API_URL=http://<your-server-host>:4200/api
- Optionally set the UI link shown in logs/CLI:
prefect config set PREFECT_UI_URL=http://<your-server-host>:4200/ui
Defaults and paths
- API default host/port: 127.0.0.1:4200 (you’ve overridden host to 0.0.0.0 to listen externally)
- API default path: /api (so the full URL is http://<host>:4200/api)
- UI default serve base: / and it calls the API at the same origin unless PREFECT_SERVER_UI_API_URL is set
Docs for reference:
- Manage settings
- Self-hosted Prefect Server (CLI)
- Settings reference
If you share the hostname/URL that external machines should use to reach your server, I can suggest the exact values to set for PREFECT_API_URL (and PREFECT_UI_URL if you want nice links).Kurt Sys (Vintecc)
10/07/2025, 4:46 PMGracefully Stopping... press Ctrl+C again to force Container agristoags-hbm-prefect-deploy-1 Stopping
Container agristoags-hbm-prefect-workpool-1 Stopping
Container agristoags-hbm-prefect-deploy-1 Stopped
prefect-workpool-1 | Traceback (most recent call last):
prefect-workpool-1 | File "/usr/local/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 44, in wrapper
prefect-workpool-1 | return fn(*args, **kwargs)
prefect-workpool-1 | File "/usr/local/lib/python3.10/site-packages/prefect/cli/_types.py", line 156, in sync_fn
prefect-workpool-1 | return asyncio.run(async_fn(*args, **kwargs))
prefect-workpool-1 | File "/usr/local/lib/python3.10/socket.py", line 967, in getaddrinfo
prefect-workpool-1 | for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
prefect-workpool-1 | socket.gaierror: [Errno -5] No address associated with hostname
prefect-workpool-1 | An exception occurred.
prefect-workpool-1 | Received SIGTERM. Sending SIGINT to the None worker (PID 1)...
prefect-workpool-1 | Received SIGINT. Sending SIGINT to the None worker (PID 1)...
prefect-workpool-1 | Worker 'DockerWorker 7624a364-d3b0-41d3-9cb7-f4704c10369b' stopped!Marvin
10/07/2025, 4:46 PMMarvin
10/07/2025, 4:47 PMPREFECT_API_URL the worker is using has a hostname it can’t resolve.
Two very likely culprits here:
1) The Slack-style angle brackets crept into your env var. If you have PREFECT_API_URL=<http://prefect:4200/api> in your compose, the worker will literally try to resolve the host “http://prefect:4200/api”, which produces socket.gaierror: [Errno -5] No address associated with hostname.
2) The hostname is not resolvable from the worker’s Docker network (e.g., using localhost, or a host-only name that’s not on the same network).
Quick fixes
- In your worker service, make sure this line has NO angle brackets:
- PREFECT_API_URL=<http://prefect:4200/api>
…and that the server service is actually named prefect and on the same Docker network.
- You do NOT need to hardcode special networks — Compose’s default service DNS works fine as long as both services share the same network.
Fast sanity checks
Run these to confirm DNS and connectivity from the worker container:
# Show what the worker thinks the API URL is
docker compose exec prefect-workpool env | grep PREFECT_API_URL
# DNS resolution for the server service name
docker compose exec prefect-workpool getent hosts prefect
# API is reachable from the worker
docker compose exec prefect-workpool curl -svf <http://prefect:4200/api/health>
# API is listening in the server container
docker compose exec prefect curl -svf <http://localhost:4200/api/health>
If those pass, prefect worker start --pool fraio should connect.
Also note: the “Gracefully Stopping.../Received SIGTERM/SIGINT” lines look like Docker Compose shutting things down after the worker crashed — they’re not the root cause. The root cause is the DNS failure.
If it still fails after removing angle brackets and confirming DNS:
- Paste the output of prefect config view from inside the worker container.
- Share the current compose environment: entries for both prefect and prefect-workpool services.
Refs:
- Self-hosted Server (Docker)
- Settings referenceKurt Sys (Vintecc)
10/07/2025, 5:27 PMMarvin
10/07/2025, 5:27 PMMarvin
10/07/2025, 5:32 PMPREFECT_API_URL inside.
Here’s a focused way to debug and fix:
Step 1: Inspect the work pool’s base job template
- This shows exactly what the worker will use when it creates the run containers.
prefect work-pool inspect fraio
Look for:
- job_configuration.env — does it include PREFECT_API_URL?
- job_configuration.networks — is your Compose network listed?
- auto_remove — set to false while debugging so containers stick around for logs.
Step 2: Attach run containers to your Compose network + set API URL
- Get your actual Docker network name (Compose rewrites it to <project>_<network>):
docker network ls | grep prefect-net
- Save the current template to a file, edit it, then update the pool:
prefect work-pool inspect fraio > docker-job.json
# edit docker-job.json:
# - Under "job_configuration", add or update:
# "networks": ["<your-actual-network-name>"],
# "env": {
# "PREFECT_API_URL": "<http://prefect:4200/api>"
# },
# "auto_remove": false,
# "stream_output": true
prefect work-pool update fraio --base-job-template ./docker-job.json
Notes:
- Use the service name prefect only if the run container is on the same network as the server container.
- Alternatively, skip networks and point to the host’s published port:
- Mac/Windows: PREFECT_API_URL=<http://host.docker.internal:4200/api>
- Linux: add an extra host mapping for “host-gateway”:
- In `job_configuration.container_create_kwargs`:
"container_create_kwargs": {
"extra_hosts": ["host.docker.internal:host-gateway"]
}
- Then set PREFECT_API_URL=<http://host.docker.internal:4200/api>
Step 3: Keep and inspect a failed run container
- With auto_remove: false, after a failure:
# Find the run container (by label)
docker ps -a --filter "label=<http://prefect.io/flow-run-id|prefect.io/flow-run-id>"
# Inspect logs and connectivity from inside that container
docker logs <container_id_or_name>
docker exec -it <container_id> sh -lc "env | grep PREFECT_API_URL; getent hosts prefect; curl -svf <http://prefect:4200/api/health>"
If DNS fails here, it confirms the run container wasn’t on the right network or had a bad API URL.
Step 4: Turn up logging on the worker
- Add to the worker service env:
PREFECT_LOGGING_LEVEL=DEBUG
- Restart the worker and watch the logs while it creates the run container; it will print the container id/name that you can target with docker logs.
Sanity checks you can run now
# From the worker container, confirm it can reach the API
docker compose exec prefect-workpool curl -svf <http://prefect:4200/api/health>
# From a throwaway container on the intended network, confirm API is reachable
docker run --rm --network <your-actual-network-name> curlimages/curl:8.10.1 -svf <http://prefect:4200/api/health>
If you want, paste:
- The output of prefect work-pool inspect fraio (redact secrets)
- The exact network name you intend to use
- A snippet of your edited docker-job.json
I’ll spot-check the template and suggest the minimal changes. References:
- Work pools and workers
- Self-hosted Server (Docker)Kurt Sys (Vintecc)
10/07/2025, 6:10 PMprefect flow-run execute 5e21d402-a9e8-489e-ac65-5c9e9c3bd12d
which results in:
18:08:58.732 | INFO | Task run 'choose_rollout-d5c' - Finished in state Completed()
18:08:58.773 | INFO | Flow run 'dangerous-pronghorn' - Finished in state Completed('All states completed.')
18:08:58.914 | INFO | prefect.flow_runs.runner - Process for flow run 'dangerous-pronghorn' exited cleanly.
18:08:58.947 | WARNING | prefect.FlowRunCancellingObserver - Consumer task exited with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/runner/_observers.py", line 176, in __aexit__
await self._consumer_task
File "/usr/local/lib/python3.10/site-packages/prefect/runner/_observers.py", line 67, in _consume_events
async for event in self._events_subscriber:
File "/usr/local/lib/python3.10/site-packages/prefect/events/clients.py", line 642, in __anext__
message = orjson.loads(await self._websocket.recv())
File "/usr/local/lib/python3.10/site-packages/websockets/asyncio/connection.py", line 322, in recv
raise self.protocol.close_exc from self.recv_exc
websockets.exceptions.ConnectionClosedError: no close frame received or sent
So, it feels like the task runs are finished, but it still somehow crashes? Also, the tasks finish too fastMarvin
10/07/2025, 6:11 PMMarvin
10/07/2025, 6:14 PMprefect flow-run inspect 5e21d402-a9e8-489e-ac65-5c9e9c3bd12d
echo $?
Exit code 0 + state Completed means the run was successful. If the warning bothers you during debugging, you can reduce log noise:
export PREFECT_LOGGING_LEVEL=ERROR
Why the worker-launched runs fail while your manual container succeeds
- When the Docker worker starts a flow-run container, that container does not inherit your compose network or environment by default. So even though your “manual” container can reach the API and run fine, the ephemeral run container may not be on the right network or may be missing PREFECT_API_URL.
Two things to set in the Docker work pool’s base job template
1) Attach the run containers to the same Docker network as your server (Compose rewrites the name to <project>_<network>).
2) Ensure PREFECT_API_URL is present inside the run containers.
How to do that
- Inspect the current template:
prefect work-pool inspect fraio
- Save, edit, and update it:
prefect work-pool inspect fraio > docker-job.json
# In docker-job.json, under "job_configuration", set:
# - use the actual network name from `docker network ls` (e.g., myproj_prefect-net)
# - set PREFECT_API_URL to a resolvable address from that network
# - keep containers for inspection while debugging
#
# {
# "job_configuration": {
# "networks": ["myproj_prefect-net"],
# "env": {
# "PREFECT_API_URL": "<http://prefect:4200/api>"
# },
# "auto_remove": false,
# "stream_output": true
# }
# }
prefect work-pool update fraio --base-job-template ./docker-job.json
Alternative (if you prefer going through the host port instead of networks)
- On Linux, add a host-gateway entry and point at host.docker.internal:
# In job_configuration:
"container_create_kwargs": {
"extra_hosts": ["host.docker.internal:host-gateway"]
},
"env": {
"PREFECT_API_URL": "<http://host.docker.internal:4200/api>"
}
Debugging a failing run container
- Keep containers: auto_remove: false
- After a failed run:
docker ps -a --filter "label=<http://prefect.io/flow-run-id|prefect.io/flow-run-id>"
docker logs <container_id>
docker exec -it <container_id> sh -lc "env | grep PREFECT_API_URL; getent hosts prefect; curl -svf <http://prefect:4200/api/health>"
This will tell you immediately if the run container can resolve/reach the API.
If you paste:
- prefect work-pool inspect fraio (redact secrets)
- The actual Docker network name from docker network ls | grep prefect
I’ll tailor the exact docker-job.json you need.
Refs
- Work pools and workers
- Self-hosted Server (Docker)Kurt Sys (Vintecc)
10/07/2025, 6:15 PMvintecc@1e9b6396f331:~$ export PREFECT_DISABLE_FLOWRUN_CANCELLING_OBSERVER=1
vintecc@1e9b6396f331:~$ prefect flow-run execute 5e21d402-a9e8-489e-ac65-5c9e9c3bd12d
18:13:09.281 | INFO | prefect.flow_runs.runner - Opening process...
18:13:11.182 | WARNING | prefect.client - Your Prefect server is running an older version of Prefect than your client which may result in unexpected behavior. Please upgrade your Prefect server from version 3.4.18 to version 3.4.22 or higher.
18:13:11.218 | INFO | prefect.engine - Finished in state Completed('All states completed.')
18:13:11.341 | INFO | prefect.flow_runs.runner - Process for flow run 'dangerous-pronghorn' exited cleanly.
so it seems there's an issue with something in the flowrun observer?Marvin
10/07/2025, 6:15 PMMarvin
10/07/2025, 6:17 PMYour Prefect server is running an older version of Prefect than your client...
Please upgrade your Prefect server from version 3.4.18 to version 3.4.22 or higher.
Valid workarounds
- Keeping PREFECT_DISABLE_FLOWRUN_CANCELLING_OBSERVER=1 is a safe way to quiet the noise. Trade-off: cancellation signals may be observed a bit later (on next API call/heartbeat), but in most flows that’s fine.
- You can also reduce noise with PREFECT_LOGGING_LEVEL=WARNING or ERROR.
Recommended fix
- Upgrade your server to match your clients/workers (3.4.22):
- If you use images, bump to prefecthq/prefect:3.4.22-python3.10 (or rebuild your image with pip install "prefect==3.4.22").
- Restart the server; it will run DB migrations automatically.
- Then re-enable the observer and confirm the warning is gone.
If you want to keep the observer disabled in worker-launched run containers, add it to the Docker work pool’s base job template:
prefect work-pool inspect fraio > docker-job.json
# In docker-job.json under "job_configuration":
# "env": {
# "PREFECT_DISABLE_FLOWRUN_CANCELLING_OBSERVER": "1",
# "... other env ..."
# }
prefect work-pool update fraio --base-job-template ./docker-job.json
Docs for reference:
- Self-hosted Server (Docker)
- Settings reference
Want me to help with the exact compose/image changes to upgrade your server to 3.4.22 so everything’s in sync?Kurt Sys (Vintecc)
10/07/2025, 6:18 PMMarvin
10/07/2025, 6:20 PMMarvin
10/07/2025, 6:21 PMimage: prefecthq/prefect:3.4.22-python3.10
- If you build your own, ensure your Dockerfile installs the matching version:
pip install "prefect==3.4.22"
- Rebuild/redeploy, then verify:
docker compose exec prefect prefect version
docker compose exec prefect prefect config view | grep PREFECT_SERVER_API_HOST
The server will automatically run DB migrations at startup.
2) Re-enable the observer and test again
- Remove PREFECT_DISABLE_FLOWRUN_CANCELLING_OBSERVER.
- Re-run:
prefect flow-run execute <run_id>
- You should see a clean completion without the websocket warning.
If you still see the warning after upgrading
- Check for any reverse proxy in front of the API/UI that might close websockets early. If you have one (e.g., Nginx/Traefik), ensure websocket upgrade headers and timeouts are set generously (e.g., Nginx proxy_read_timeout 3600; and proper Connection/Upgrade headers).
- Turn on debug logs to see observer details:
export PREFECT_LOGGING_LEVEL=DEBUG
prefect flow-run execute <run_id>
- Look at server logs around the time the run finishes to see if the WS is being closed by the server.
If you want, share:
- How you’re building the server image (Dockerfile snippet or exact image tag)
- Whether you have a reverse proxy in front of Prefect
I can give you the exact update steps and proxy settings to ensure the observer exits cleanly.Kurt Sys (Vintecc)
10/07/2025, 7:07 PMMarvin
10/07/2025, 7:08 PMMarvin
10/07/2025, 7:08 PM# If you build your own image
pip install "prefect==3.4.22"
# Or use the official image tag
prefecthq/prefect:3.4.22-python3.10
References:
- Self-hosted Server (Docker)
- Settings reference