<@ULVA73B9P> I'm having a setup in docker, where I...
# ask-marvin
k
@Marvin I'm having a setup in docker, where I start a prefect server, and I setup a worker (in docker) and init deployment (in docker). Everything seems to work, until I start a run, which makes it crash. I seem to have some issues with network:
Copy code
File "/usr/local/lib/python3.10/socket.py", line 967, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -5] No address associated with hostname An exception occurred.
So, although I have a perfectly fine running work pool and worker, when triggering a run, it seems the address/hostname is wrong
m
thought for 439.7 seconds
It sounds like your flow-run container can’t resolve the host in
PREFECT_API_URL
. In Docker,
localhost
points to the container itself, and service names only resolve on the same user-defined network. Your worker can reach the API (polling works) but the flow-run container (spawned by the docker worker) likely isn’t on the same network or is given an API URL that isn’t resolvable. Fix options (pick one): 1) Use your Compose network + service name (recommended) - Put your Prefect Server and Worker services on a named user-defined network (e.g.,
prefect-net
). - Make sure the docker worker launches flow-run containers on that same network via the work pool’s base job template. - Set
PREFECT_API_URL
to the server’s service name on that network (e.g., `http://prefect-server:4200/api`). Example docker-compose.yml (abridged):
Copy code
networks:
  prefect-net:

services:
  prefect-server:
    image: prefecthq/prefect:3-latest
    command: prefect server start
    networks: [prefect-net]
    # expose/mapping: "4200:4200" for host access
  prefect-worker:
    image: prefecthq/prefect:3-latest
    command: prefect worker start -p docker-pool
    environment:
      PREFECT_API_URL: <http://prefect-server:4200/api>
    networks: [prefect-net]
Create/update your docker work pool so flow-run containers join that same network:
Copy code
prefect work-pool get-default-base-job-template --type docker --file base-template.json

# Edit to include at least:
# {
#   "networks": ["prefect-net"],
#   "env": {"PREFECT_API_URL": "<http://prefect-server:4200/api>"}
# }

prefect work-pool create docker-pool --type docker --base-job-template base-template.json --overwrite
2) Use the host gateway (Linux) with host.docker.internal - If your server is on the host, set
PREFECT_API_URL=<http://host.docker.internal:4200/api>
. - Ensure flow-run containers can resolve
host.docker.internal
by adding an extra host mapping via the work pool template:
Copy code
{
  "env": {
    "PREFECT_API_URL": "<http://host.docker.internal:4200/api>"
  },
  "container_create_kwargs": {
    "extra_hosts": {
      "host.docker.internal": "host-gateway"
    }
  }
}
3) Use your host’s LAN IP - Set
PREFECT_API_URL
to
http://<host-ip>:4200/api
. Ensure networking allows access from containers. Why this happens - The worker passes its Prefect settings (including
PREFECT_API_URL
) into the flow-run container. If that hostname isn’t resolvable from the flow-run container’s network context, you’ll see
socket.gaierror: [Errno -5] No address associated with hostname
. Quick checks - In the worker and flow-run containers, run:
Copy code
prefect config view | grep PREFECT_API_URL
- From a temporary container on the intended network, verify DNS:
Copy code
docker run --rm --network prefect-net alpine sh -c "apk add --no-cache bind-tools >/dev/null && nslookup prefect-server"
If you’d like, share: - Your
docker-compose.yml
(services + networks section) - The work pool name and its base job template (
prefect work-pool inspect <pool>
) - What
PREFECT_API_URL
resolves to in your worker container Related docs - Self-hosted Prefect Server with Docker - Manage settings (PREFECT_API_URL) - Blog: Work pools and workers overview
k
@Marvin so it makes sense to put
networks
in every container, so they can find each other and I have the hard coded value? I can add multiple networks for each service, and it will look in all these networks for the right container? btw, I'm using prefect 3.x?
m
thought for 33.2 seconds
k
@Marvin I can't use something like this?
Copy code
deployment_id = trigger_flow.deploy(
        name="fraio-trigger",
        work_pool_name="fraio",
        image="harbor.mgmt.vintecc.cloud/vintecc-ags-hbm/fraio_algo:local",
        build=False,
        push=False,
        version="local",
        tags=[],
        description=None,
        entrypoint_type=EntrypointType.MODULE_PATH,
        concurrency_limit=5,
        print_next_steps=True,
        infrastructure_overrides={
            "env": {
                "PREFECT_API_URL": "<http://prefect:4200/api>",
            },
            "network": "agristoags-hbm_default",  # ensure same network
        },
    )
m
thought for 89.2 seconds
Great questions — and yes, you’re on Prefect 3.x (I see 3.4.22). - Does it make sense to put services on a shared network and use a “hardcoded” service name? Yes. In Docker, containers on the same user-defined network can reach each other by service name (e.g.,
prefect-server
). Using the service name in
PREFECT_API_URL
is the right approach. - Can I attach multiple networks and it will “look in all of them”? Not exactly. Docker’s DNS will only resolve names for containers that share a network with the caller. If two containers share multiple networks, the name still resolves (to the peer’s IP on one of the shared networks). Docker does not “bridge” networks for you. Practically: - Ensure the flow-run container and the Prefect Server share at least one network. - To avoid ambiguity and surprises, prefer a single shared network for all Prefect services and flow-run containers (e.g.,
prefect-net
). Important Prefect-specific note - Your flow runs are launched by the Docker worker as new containers outside docker-compose. They won’t be on your compose network unless you tell the work pool to attach them. Set
networks
in the Docker work pool’s base job template so every flow-run container joins the same network as your server. Example base job template for the Docker work pool
Copy code
{
  "image": "your/image:tag",
  "env": {
    "PREFECT_API_URL": "<http://prefect-server:4200/api>"
  },
  "networks": [
    "prefect-net"
  ]
}
How to apply the base job template
Copy code
prefect work-pool get-default-base-job-template --type docker --file base-template.json
# edit base-template.json as above
prefect work-pool create docker-pool --type docker --base-job-template base-template.json --overwrite
Example docker-compose sketch
Copy code
networks:
  prefect-net:

services:
  prefect-server:
    image: prefecthq/prefect:3-latest
    command: prefect server start
    networks: [prefect-net]
    ports:
      - "4200:4200"

  prefect-worker:
    image: prefecthq/prefect:3-latest
    command: prefect worker start -p docker-pool
    environment:
      PREFECT_API_URL: <http://prefect-server:4200/api>
    networks: [prefect-net]
Alternative if your server runs on the host (not in Docker) on Linux - Use the host gateway mapping so containers can reach the host:
Copy code
{
  "env": {
    "PREFECT_API_URL": "<http://host.docker.internal:4200/api>"
  },
  "container_create_kwargs": {
    "extra_hosts": {
      "host.docker.internal": "host-gateway"
    }
  }
}
Quick sanity checks - In both your worker container and any flow-run container:
Copy code
prefect config view | grep PREFECT_API_URL
- DNS on the network:
Copy code
docker run --rm --network prefect-net alpine sh -c "apk add --no-cache bind-tools >/dev/null && nslookup prefect-server"
Docs and references - Self-hosted Prefect Server with Docker - Manage settings (PREFECT_API_URL) If you want, share your compose file (services + networks) and the work pool’s base job template (
prefect work-pool inspect <pool>
), and I’ll help you tighten it up.
You’re very close, but a couple of important tweaks are needed in Prefect 3.x: - Use
job_variables
(not
infrastructure_overrides
) - For Docker, the field is
networks
(list), not
network
- Make sure
PREFECT_API_URL
is a plain URL string without angle brackets Corrected example:
Copy code
deployment_id = trigger_flow.deploy(
    name="fraio-trigger",
    work_pool_name="fraio",  # must be a Docker work pool
    image="harbor.mgmt.vintecc.cloud/vintecc-ags-hbm/fraio_algo:local",
    build=False,
    push=False,
    version="local",
    entrypoint_type=EntrypointType.MODULE_PATH,
    concurrency_limit=5,
    print_next_steps=True,
    job_variables={
        "env": {
            "PREFECT_API_URL": "<http://prefect:4200/api>"
        },
        "networks": ["agristoags-hbm_default"],  # ensure same network as your server service
        # optional: if your server is on the host instead of the same network:
        # "container_create_kwargs": {
        #     "extra_hosts": {"host.docker.internal": "host-gateway"}
        # }
    },
)
Why this matters: - The docker worker creates a new container for each flow run. That container must be able to resolve the Prefect API hostname. Using the same user-defined network and a service name (e.g.,
prefect
) is the most reliable approach. -
socket.gaierror: [Errno -5]
often appears when the hostname in
PREFECT_API_URL
is not resolvable from the flow-run container. Angle brackets in the URL (e.g.,
"<http://prefect:4200/api>"
) will also break DNS resolution — use
<http://prefect:4200/api>
. Quick verifications: - Confirm your work pool is Docker:
prefect work-pool inspect fraio
- See allowed job variables:
prefect work-pool get-default-base-job-template --type docker
- In a running flow-run container, check settings:
prefect config view | grep PREFECT_API_URL
If you still hit issues, please share: - The
prefect work-pool inspect fraio
output - The service and network names in your docker-compose.yml References: - Customize job variables - Manage settings (PREFECT_API_URL) - Self-hosted Prefect Server with Docker
k
@Marvin what about this:
Copy code
networks:
  prefect-net:

services:
  prefect:
    build:
      context: .
      dockerfile: Dockerfile.prefect
    container_name: prefect
    restart: always
    volumes:
      - prefect:/root/.prefect
    entrypoint: ["/opt/prefect/entrypoint.sh", "prefect", "server", "start"]
    environment:
      - PREFECT_UI_URL=<http://127.0.0.1:4200/ui>
      - PREFECT_SERVER_API_HOST=0.0.0.0
      - PREFECT_API_DATABASE_CONNECTION_URL=postgresql+asyncpg://${ORCHESTRATOR_DB_USER}:${ORCHESTRATOR_DB_PASSWORD}@${ORCHESTRATOR_DB_HOST}:${ORCHESTRATOR_DB_PORT}/${ORCHESTRATOR_DB_NAME}
    ports:
      - 4200:4200
    depends_on:
      - fraio-data
    healthcheck:
      test: ["CMD", "curl", "-fsS", "<http://localhost:4200/api/health>"]
      interval: 5s
      timeout: 3s
      retries: 20
    networks: [prefect-net]
    profiles:
      - dev

  prefect-workpool:
    build:
      context: .
      dockerfile: Dockerfile.prefect-workpool
    depends_on:
      prefect:
        condition: service_healthy
    environment:
      - PREFECT_API_URL=<http://prefect:4200/api>
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    entrypoint: ["/bin/bash", "-c"]
    command: |
      "
      set -e

      echo 'Creating work pool...'
      prefect work-pool create fraio -t docker --overwrite

      echo 'Starting worker for fraio...'
      prefect worker start --pool 'fraio'
      "
    networks: [prefect-net]
    profiles:
      - dev
m
thought for 333.3 seconds
k
@Marvin btw, I have this in my prefect server:
Copy code
root@9bf7f4cf741d:/opt/prefect# prefect config view | grep PREFECT_API_URL
root@9bf7f4cf741d:/opt/prefect# prefect config view 
🚀 you are connected to:
<http://127.0.0.1:4200/ui>
PREFECT_PROFILE='ephemeral'
PREFECT_API_DATABASE_CONNECTION_URL='********' (from env)
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='true' (from profile)
PREFECT_SERVER_API_HOST='0.0.0.0' (from env)
PREFECT_UI_URL='<http://127.0.0.1:4200/ui>' (from env)
m
thought for 289.2 seconds
Thanks for sharing the compose snippet! Overall you’re very close. Here’s what looks good, what to tweak, and a cleaned-up example. What’s correct - Using Postgres via
PREFECT_API_DATABASE_CONNECTION_URL=postgresql+asyncpg://...
is supported in Prefect 3.x. Prefect will auto-create the schema on first start. - Exposing the server on 0.0.0.0 is supported.
PREFECT_SERVER_API_HOST
maps to the
--host
flag for
prefect server start
(confirmed by
prefect server start --help
). -
PREFECT_UI_URL
is the right setting to advertise an externally reachable UI URL. - Docker worker setup: -
prefect work-pool create fraio -t docker --overwrite
is valid. -
prefect worker start --pool fraio
is valid. - Mounting
/var/run/docker.sock
is required for the Docker worker. - Pointing the worker at the API with
PREFECT_API_URL=<http://prefect:4200/api>
(using the compose service name) is correct. - Healthcheck against
<http://localhost:4200/api/health>
is appropriate. Things to fix or double-check - Remove Slack-style angle brackets from URLs in env vars. In compose they’ll be literal. For example: -
PREFECT_UI_URL=<http://127.0.0.1:4200/ui>
-
PREFECT_API_URL=<http://prefect:4200/api>
- Ensure the image you build actually has
/opt/prefect/entrypoint.sh
. That script exists in the official Prefect images; if you’re building your own, either copy it or switch to a
command: ["prefect","server","start"]
style. - Your worker
command
is wrapped in extra quotes. Those quotes will be passed to bash and can break the script. Remove the outer quotes. - Make sure
curl
is present in the server image if you keep the curl-based healthcheck. - You’re using a named volume
prefect:/root/.prefect
but didn’t declare it at the top-level. Add a
volumes:
section. - If your DB is a separate compose service, add a proper
depends_on
(with a healthcheck) to ensure Postgres is ready before the API starts. A tightened compose example
Copy code
yaml
networks:
  prefect-net:

volumes:
  prefect:

services:
  prefect:
    # If you prefer, you can use the official image instead of build:
    # image: prefecthq/prefect:3-python3.10
    build:
      context: .
      dockerfile: Dockerfile.prefect
    container_name: prefect
    restart: unless-stopped
    volumes:
      - prefect:/root/.prefect
    # If your image includes the official entrypoint script, keep this.
    # Otherwise, prefer the command below.
    # entrypoint: ["/opt/prefect/entrypoint.sh", "prefect", "server", "start"]
    command: ["prefect", "server", "start", "--host", "0.0.0.0", "--port", "4200"]
    environment:
      - PREFECT_UI_URL=<http://127.0.0.1:4200/ui>
      - PREFECT_SERVER_API_HOST=0.0.0.0
      - PREFECT_API_DATABASE_CONNECTION_URL=postgresql+asyncpg://${ORCHESTRATOR_DB_USER}:${ORCHESTRATOR_DB_PASSWORD}@${ORCHESTRATOR_DB_HOST}:${ORCHESTRATOR_DB_PORT}/${ORCHESTRATOR_DB_NAME}
    ports:
      - "4200:4200"
    # If you have a Postgres service, add:
    # depends_on:
    #   postgres:
    #     condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-fsS", "<http://localhost:4200/api/health>"]
      interval: 5s
      timeout: 3s
      retries: 20
    networks: [prefect-net]
    profiles: [dev]

  prefect-workpool:
    # image: prefecthq/prefect:3-python3.10
    build:
      context: .
      dockerfile: Dockerfile.prefect-workpool
    depends_on:
      prefect:
        condition: service_healthy
    environment:
      - PREFECT_API_URL=<http://prefect:4200/api>
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: unless-stopped
    entrypoint: ["/bin/bash", "-lc"]
    command: |
      set -e
      echo 'Creating work pool...'
      prefect work-pool create fraio --type docker --overwrite
      echo 'Starting worker for fraio...'
      prefect worker start --pool fraio
    networks: [prefect-net]
    profiles: [dev]
Notes
- If you’re reverse-proxying the UI (e.g., behind Nginx at https://prefect.example.com), set
PREFECT_UI_URL
to that public URL. - Switching from SQLite to Postgres creates a new, empty API DB; Prefect will migrate/create tables automatically. - You can optionally set a worker name with
--name
for easier identification. Docs - Server via Docker: Self-hosted Prefect Server (Docker) - Settings reference: Settings reference - Work pools and workers overview: Work pools and workers Questions for you - Are you basing your images on the official Prefect image? If not, do you want to switch to
command: ["prefect", "server", "start", ...]
instead of relying on
/opt/prefect/entrypoint.sh
? - Do you want a sample Postgres service and healthcheck wired into
depends_on
?
Thanks for sharing that output — it helps. Here’s how these settings work in Prefect 3.x and why you don’t see
PREFECT_API_URL
in your server container: -
PREFECT_API_URL
(client-side): This tells Prefect clients (CLI, workers, flow code) where to reach the API. It should point at your server’s externally reachable API endpoint, typically
http://<host>:4200/api
. You set this on machines running workers/CLI/flows — not in the server container — so it’s normal that
prefect config view
inside the server does not show it unless you’ve explicitly set it there. - `PREFECT_SERVER_API_HOST`/`PORT`/`BASE_PATH` (server-side): Controls how the API binds inside the server. Defaults are host
127.0.0.1
, port
4200
, and base path unset. You’ve set
PREFECT_SERVER_API_HOST=0.0.0.0
, which is correct to listen on all interfaces. Make sure you expose/map port 4200 from your container/host so clients can reach it. -
PREFECT_UI_URL
(client convenience): Used by CLI/clients to print a clickable UI link. It doesn’t affect API connectivity. Set this on your workers/CLI to your externally reachable UI URL (e.g.,
http://<host>:4200/ui
) so “View in UI” links are correct. - UI → API connection: The server’s UI will call the API at the same origin by default (i.e., it assumes
/api
on the same host/port it’s served from). You can override it with
PREFECT_SERVER_UI_API_URL
if you’re fronting the UI/API with a proxy or using nonstandard paths. Otherwise you usually don’t need to set this. What to set so external workers/CLI can connect - On each worker/CLI machine (or in your worker container/pod environment), set:
Copy code
export PREFECT_API_URL=http://<your-server-host>:4200/api
Or persist it in the active profile:
Copy code
prefect config set PREFECT_API_URL=http://<your-server-host>:4200/api
- Optionally set the UI link shown in logs/CLI:
Copy code
prefect config set PREFECT_UI_URL=http://<your-server-host>:4200/ui
Defaults and paths - API default host/port: 127.0.0.1:4200 (you’ve overridden host to 0.0.0.0 to listen externally) - API default path:
/api
(so the full URL is
http://<host>:4200/api
) - UI default serve base:
/
and it calls the API at the same origin unless
PREFECT_SERVER_UI_API_URL
is set Docs for reference: - Manage settings - Self-hosted Prefect Server (CLI) - Settings reference If you share the hostname/URL that external machines should use to reach your server, I can suggest the exact values to set for
PREFECT_API_URL
(and
PREFECT_UI_URL
if you want nice links).
k
@Marvin it all doesn't make any sense to me: I can run it all perfectly fine from the command line. I just moved the commands from the command line to a docker container, and the workers fail to run the runs/workflows. So I changed/hard coded the networks, but it still fails, and now:
Copy code
Gracefully Stopping... press Ctrl+C again to force Container agristoags-hbm-prefect-deploy-1  Stopping
 Container agristoags-hbm-prefect-workpool-1  Stopping
 Container agristoags-hbm-prefect-deploy-1  Stopped
prefect-workpool-1  | Traceback (most recent call last):
prefect-workpool-1  |   File "/usr/local/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 44, in wrapper
prefect-workpool-1  |     return fn(*args, **kwargs)
prefect-workpool-1  |   File "/usr/local/lib/python3.10/site-packages/prefect/cli/_types.py", line 156, in sync_fn
prefect-workpool-1  |     return asyncio.run(async_fn(*args, **kwargs))

prefect-workpool-1  |   File "/usr/local/lib/python3.10/socket.py", line 967, in getaddrinfo
prefect-workpool-1  |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
prefect-workpool-1  | socket.gaierror: [Errno -5] No address associated with hostname
prefect-workpool-1  | An exception occurred.
prefect-workpool-1  | Received SIGTERM. Sending SIGINT to the None worker (PID 1)...
prefect-workpool-1  | Received SIGINT. Sending SIGINT to the None worker (PID 1)...
prefect-workpool-1  | Worker 'DockerWorker 7624a364-d3b0-41d3-9cb7-f4704c10369b' stopped!
m
thought for 77.8 seconds
I hear you — that error is super frustrating when everything worked on the host. The stacktrace you shared points to a DNS resolution failure inside the worker container. In practice, this almost always means the
PREFECT_API_URL
the worker is using has a hostname it can’t resolve. Two very likely culprits here: 1) The Slack-style angle brackets crept into your env var. If you have
PREFECT_API_URL=<http://prefect:4200/api>
in your compose, the worker will literally try to resolve the host “http://prefect:4200/api”, which produces
socket.gaierror: [Errno -5] No address associated with hostname
. 2) The hostname is not resolvable from the worker’s Docker network (e.g., using
localhost
, or a host-only name that’s not on the same network). Quick fixes - In your worker service, make sure this line has NO angle brackets:
Copy code
- PREFECT_API_URL=<http://prefect:4200/api>
…and that the server service is actually named
prefect
and on the same Docker network. - You do NOT need to hardcode special networks — Compose’s default service DNS works fine as long as both services share the same network. Fast sanity checks Run these to confirm DNS and connectivity from the worker container:
Copy code
# Show what the worker thinks the API URL is
docker compose exec prefect-workpool env | grep PREFECT_API_URL

# DNS resolution for the server service name
docker compose exec prefect-workpool getent hosts prefect

# API is reachable from the worker
docker compose exec prefect-workpool curl -svf <http://prefect:4200/api/health>

# API is listening in the server container
docker compose exec prefect curl -svf <http://localhost:4200/api/health>
If those pass,
prefect worker start --pool fraio
should connect. Also note: the “Gracefully Stopping.../Received SIGTERM/SIGINT” lines look like Docker Compose shutting things down after the worker crashed — they’re not the root cause. The root cause is the DNS failure. If it still fails after removing angle brackets and confirming DNS: - Paste the output of
prefect config view
from inside the worker container. - Share the current compose
environment:
entries for both
prefect
and
prefect-workpool
services. Refs: - Self-hosted Server (Docker) - Settings reference
k
@Marvin naaa, networking seems to be fine, until an actual worker/runner container is started. There, something fails, but I have idea how to debug it
m
thought for 332.6 seconds
Got it — if DNS looks fine until the worker launches the actual flow-run container, the issue is almost certainly in how that ephemeral container is created. By default, the Docker worker creates sibling containers on the host via the Docker socket; they won’t inherit your Compose network unless you explicitly attach them, and they may not have the right
PREFECT_API_URL
inside. Here’s a focused way to debug and fix: Step 1: Inspect the work pool’s base job template - This shows exactly what the worker will use when it creates the run containers.
Copy code
prefect work-pool inspect fraio
Look for: -
job_configuration.env
— does it include
PREFECT_API_URL
? -
job_configuration.networks
— is your Compose network listed? -
auto_remove
— set to false while debugging so containers stick around for logs. Step 2: Attach run containers to your Compose network + set API URL - Get your actual Docker network name (Compose rewrites it to <project>_<network>):
Copy code
docker network ls | grep prefect-net
- Save the current template to a file, edit it, then update the pool:
Copy code
prefect work-pool inspect fraio > docker-job.json
# edit docker-job.json:
#  - Under "job_configuration", add or update:
#      "networks": ["<your-actual-network-name>"],
#      "env": {
#        "PREFECT_API_URL": "<http://prefect:4200/api>"
#      },
#      "auto_remove": false,
#      "stream_output": true
prefect work-pool update fraio --base-job-template ./docker-job.json
Notes: - Use the service name
prefect
only if the run container is on the same network as the server container. - Alternatively, skip networks and point to the host’s published port: - Mac/Windows:
PREFECT_API_URL=<http://host.docker.internal:4200/api>
- Linux: add an extra host mapping for “host-gateway”: - In `job_configuration.container_create_kwargs`:
Copy code
"container_create_kwargs": {
        "extra_hosts": ["host.docker.internal:host-gateway"]
      }
- Then set
PREFECT_API_URL=<http://host.docker.internal:4200/api>
Step 3: Keep and inspect a failed run container - With
auto_remove: false
, after a failure:
Copy code
# Find the run container (by label)
docker ps -a --filter "label=<http://prefect.io/flow-run-id|prefect.io/flow-run-id>"

# Inspect logs and connectivity from inside that container
docker logs <container_id_or_name>
docker exec -it <container_id> sh -lc "env | grep PREFECT_API_URL; getent hosts prefect; curl -svf <http://prefect:4200/api/health>"
If DNS fails here, it confirms the run container wasn’t on the right network or had a bad API URL. Step 4: Turn up logging on the worker - Add to the worker service env:
Copy code
PREFECT_LOGGING_LEVEL=DEBUG
- Restart the worker and watch the logs while it creates the run container; it will print the container id/name that you can target with
docker logs
. Sanity checks you can run now
Copy code
# From the worker container, confirm it can reach the API
docker compose exec prefect-workpool curl -svf <http://prefect:4200/api/health>

# From a throwaway container on the intended network, confirm API is reachable
docker run --rm --network <your-actual-network-name> curlimages/curl:8.10.1 -svf <http://prefect:4200/api/health>
If you want, paste: - The output of
prefect work-pool inspect fraio
(redact secrets) - The exact network name you intend to use - A snippet of your edited
docker-job.json
I’ll spot-check the template and suggest the minimal changes. References: - Work pools and workers - Self-hosted Server (Docker)
k
@Marvin let's try another way: inside my container with the run code, I do run:
Copy code
prefect flow-run execute 5e21d402-a9e8-489e-ac65-5c9e9c3bd12d
which results in:
Copy code
18:08:58.732 | INFO    | Task run 'choose_rollout-d5c' - Finished in state Completed()
18:08:58.773 | INFO    | Flow run 'dangerous-pronghorn' - Finished in state Completed('All states completed.')
18:08:58.914 | INFO    | prefect.flow_runs.runner - Process for flow run 'dangerous-pronghorn' exited cleanly.
18:08:58.947 | WARNING | prefect.FlowRunCancellingObserver - Consumer task exited with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/runner/_observers.py", line 176, in __aexit__
    await self._consumer_task
  File "/usr/local/lib/python3.10/site-packages/prefect/runner/_observers.py", line 67, in _consume_events
    async for event in self._events_subscriber:
  File "/usr/local/lib/python3.10/site-packages/prefect/events/clients.py", line 642, in __anext__
    message = orjson.loads(await self._websocket.recv())
  File "/usr/local/lib/python3.10/site-packages/websockets/asyncio/connection.py", line 322, in recv
    raise self.protocol.close_exc from self.recv_exc
websockets.exceptions.ConnectionClosedError: no close frame received or sent
So, it feels like the task runs are finished, but it still somehow crashes? Also, the tasks finish too fast
m
thought for 184.3 seconds
Thanks for the extra detail — that helps narrow it down. What you’re seeing from prefect flow-run execute is not a crash of your flow. The important lines are: - “Flow run … Finished in state Completed” - Then a WARNING from FlowRunCancellingObserver with websockets.exceptions.ConnectionClosedError That warning appears when the flow process tears down its event subscription (websocket) as the process exits. In Docker, the process often exits quickly or the container is being stopped, so the websocket closes without a “close frame” and the observer logs a warning during shutdown. It’s noisy but harmless if your run already shows Completed. You can confirm:
Copy code
prefect flow-run inspect 5e21d402-a9e8-489e-ac65-5c9e9c3bd12d
echo $?
Exit code 0 + state Completed means the run was successful. If the warning bothers you during debugging, you can reduce log noise:
Copy code
export PREFECT_LOGGING_LEVEL=ERROR
Why the worker-launched runs fail while your manual container succeeds - When the Docker worker starts a flow-run container, that container does not inherit your compose network or environment by default. So even though your “manual” container can reach the API and run fine, the ephemeral run container may not be on the right network or may be missing PREFECT_API_URL. Two things to set in the Docker work pool’s base job template 1) Attach the run containers to the same Docker network as your server (Compose rewrites the name to <project>_<network>). 2) Ensure PREFECT_API_URL is present inside the run containers. How to do that - Inspect the current template:
Copy code
prefect work-pool inspect fraio
- Save, edit, and update it:
Copy code
prefect work-pool inspect fraio > docker-job.json

# In docker-job.json, under "job_configuration", set:
# - use the actual network name from `docker network ls` (e.g., myproj_prefect-net)
# - set PREFECT_API_URL to a resolvable address from that network
# - keep containers for inspection while debugging
#
# {
#   "job_configuration": {
#     "networks": ["myproj_prefect-net"],
#     "env": {
#       "PREFECT_API_URL": "<http://prefect:4200/api>"
#     },
#     "auto_remove": false,
#     "stream_output": true
#   }
# }

prefect work-pool update fraio --base-job-template ./docker-job.json
Alternative (if you prefer going through the host port instead of networks) - On Linux, add a host-gateway entry and point at host.docker.internal:
Copy code
# In job_configuration:
"container_create_kwargs": {
  "extra_hosts": ["host.docker.internal:host-gateway"]
},
"env": {
  "PREFECT_API_URL": "<http://host.docker.internal:4200/api>"
}
Debugging a failing run container - Keep containers:
auto_remove: false
- After a failed run:
Copy code
docker ps -a --filter "label=<http://prefect.io/flow-run-id|prefect.io/flow-run-id>"
docker logs <container_id>
docker exec -it <container_id> sh -lc "env | grep PREFECT_API_URL; getent hosts prefect; curl -svf <http://prefect:4200/api/health>"
This will tell you immediately if the run container can resolve/reach the API. If you paste: -
prefect work-pool inspect fraio
(redact secrets) - The actual Docker network name from
docker network ls | grep prefect
I’ll tailor the exact
docker-job.json
you need. Refs - Work pools and workers - Self-hosted Server (Docker)
k
@Marvin I got it working like this:
Copy code
vintecc@1e9b6396f331:~$ export PREFECT_DISABLE_FLOWRUN_CANCELLING_OBSERVER=1
vintecc@1e9b6396f331:~$ prefect flow-run execute 5e21d402-a9e8-489e-ac65-5c9e9c3bd12d
18:13:09.281 | INFO    | prefect.flow_runs.runner - Opening process...
18:13:11.182 | WARNING | prefect.client - Your Prefect server is running an older version of Prefect than your client which may result in unexpected behavior. Please upgrade your Prefect server from version 3.4.18 to version 3.4.22 or higher.
18:13:11.218 | INFO    | prefect.engine - Finished in state Completed('All states completed.')
18:13:11.341 | INFO    | prefect.flow_runs.runner - Process for flow run 'dangerous-pronghorn' exited cleanly.
so it seems there's an issue with something in the flowrun observer?
m
thought for 82.3 seconds
Nice find — that lines up with what we were seeing. What’s happening - Your flow actually completes successfully. The warning is from the FlowRunCancellingObserver tearing down its events websocket as the process exits. If the process/container stops quickly, the websocket can close without a “close frame,” which triggers that warning during shutdown. - You also have a version mismatch: server 3.4.18 vs client 3.4.22. There have been changes around events/websocket handling; mismatched versions can make these shutdown messages more likely:
Copy code
Your Prefect server is running an older version of Prefect than your client...
  Please upgrade your Prefect server from version 3.4.18 to version 3.4.22 or higher.
Valid workarounds - Keeping
PREFECT_DISABLE_FLOWRUN_CANCELLING_OBSERVER=1
is a safe way to quiet the noise. Trade-off: cancellation signals may be observed a bit later (on next API call/heartbeat), but in most flows that’s fine. - You can also reduce noise with
PREFECT_LOGGING_LEVEL=WARNING
or
ERROR
. Recommended fix - Upgrade your server to match your clients/workers (3.4.22): - If you use images, bump to
prefecthq/prefect:3.4.22-python3.10
(or rebuild your image with
pip install "prefect==3.4.22"
). - Restart the server; it will run DB migrations automatically. - Then re-enable the observer and confirm the warning is gone. If you want to keep the observer disabled in worker-launched run containers, add it to the Docker work pool’s base job template:
Copy code
prefect work-pool inspect fraio > docker-job.json
# In docker-job.json under "job_configuration":
# "env": {
#   "PREFECT_DISABLE_FLOWRUN_CANCELLING_OBSERVER": "1",
#   "... other env ..."
# }
prefect work-pool update fraio --base-job-template ./docker-job.json
Docs for reference: - Self-hosted Server (Docker) - Settings reference Want me to help with the exact compose/image changes to upgrade your server to 3.4.22 so everything’s in sync?
k
@Marvin well, I'd prefer to have the flowrun observer working fine?
m
thought for 45.0 seconds
Totally fair — let’s make the observer work, not hide it. What’s going on - Your flow finishes fine; the warning is from the FlowRunCancellingObserver as it shuts down its events websocket. - You’re seeing a version mismatch: server 3.4.18 vs client 3.4.22. We’ve seen websocket shutdown warnings like this when versions differ. Matching versions generally resolves it. What I recommend 1) Upgrade the server to match 3.4.22 - If you use the official image, bump it to something like:
Copy code
image: prefecthq/prefect:3.4.22-python3.10
- If you build your own, ensure your Dockerfile installs the matching version:
Copy code
pip install "prefect==3.4.22"
- Rebuild/redeploy, then verify:
Copy code
docker compose exec prefect prefect version
  docker compose exec prefect prefect config view | grep PREFECT_SERVER_API_HOST
The server will automatically run DB migrations at startup. 2) Re-enable the observer and test again - Remove
PREFECT_DISABLE_FLOWRUN_CANCELLING_OBSERVER
. - Re-run:
Copy code
prefect flow-run execute <run_id>
- You should see a clean completion without the websocket warning. If you still see the warning after upgrading - Check for any reverse proxy in front of the API/UI that might close websockets early. If you have one (e.g., Nginx/Traefik), ensure websocket upgrade headers and timeouts are set generously (e.g., Nginx
proxy_read_timeout 3600;
and proper
Connection/Upgrade
headers). - Turn on debug logs to see observer details:
Copy code
export PREFECT_LOGGING_LEVEL=DEBUG
  prefect flow-run execute <run_id>
- Look at server logs around the time the run finishes to see if the WS is being closed by the server. If you want, share: - How you’re building the server image (Dockerfile snippet or exact image tag) - Whether you have a reverse proxy in front of Prefect I can give you the exact update steps and proxy settings to ensure the observer exits cleanly.
k
@Marvin just to be sure: having version 3.4.18 and 3.4.22 seem to be incompatible somehow?
m
thought for 22.9 seconds
Short answer: not “hard” incompatible, but mismatched versions can cause exactly the kind of oddities you saw. - Prefect 3.x aims for compatibility across patch versions, but if your client/worker is newer than the server, new client features can hit older server behavior. We warn in that case because it can lead to unexpected behavior (e.g., the FlowRunCancellingObserver’s websocket shutdown you hit between 3.4.18 and 3.4.22). - Core scheduling/execution will usually still work, but you may see noisy warnings or subtle issues. Best practice: - Keep server >= clients/workers. Either upgrade the server to 3.4.22, or pin clients/workers to 3.4.18 to match. - After aligning, re-enable the observer and re-test. Quick options:
Copy code
# If you build your own image
pip install "prefect==3.4.22"

# Or use the official image tag
prefecthq/prefect:3.4.22-python3.10
References: - Self-hosted Server (Docker) - Settings reference
133 Views