Hello all, I’m having some difficulty in having the agent connect to the server when running deployments with the docker infrastructure block , but not when running flows directly. Some context: • I’m using docker compose to spin up a

prefect-server

and

prefect-agent

making sure they are in the same named network. • The agent has the host’s docker socket mounted in volumes so it can spin up parallel containers with the Docker infrastructure block, not via Docker-in-Docker. • Created a custom flow runner image based on the official docker image with flows installed in

/opt/prefect/flows

and all the python dependencies installed. • I have created a Docker infrastructure block from the host machine to have the agent spin up the flow runner containers. After I have registered both the Docker block and a deployment, I have tried to do a quick run of the deployment through the UI, but the flow run crashes. The agent log shows (relevant bits):

12:59:26.941 | INFO    | prefect.agent - Submitting flow run '977526d8-f473-4713-b454-a792df88af5f'
12:59:27.023 | INFO    | prefect.infrastructure.docker-container - Creating Docker container 'meteoric-bird'...
12:59:27.160 | INFO    | prefect.infrastructure.docker-container - Docker container 'meteoric-bird' has status 'created'
12:59:27.450 | INFO    | prefect.agent - Completed submission of flow run '977526d8-f473-4713-b454-a792df88af5f'
12:59:27.453 | INFO    | prefect.infrastructure.docker-container - Docker container 'meteoric-bird' has status 'running'
17:59:28.323 | DEBUG   | prefect.profiles - Using profile 'default'
<frozen runpy>:128: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
17:59:28.335 | DEBUG   | prefect.client - Connecting to API at <http://prefect-server:4200/api/>
17:59:29.873 | ERROR   | prefect.engine - Engine execution of flow run '977526d8-f473-4713-b454-a792df88af5f' exited with unexpected exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 186, in connect_tcp
    addr_obj = ip_address(remote_host)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ipaddress.py", line 54, in ip_address
    raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
ValueError: 'prefect-server' does not appear to be an IPv4 or IPv6 address

Some things I’ve tried or confirmed: • The

prefect-agent

and

prefect-server

containers are on the same network. Confirmed by

docker network inspect

. • The

prefect-agent

can see the

prefect-server

container and resolve the name when I use both ping (

ping prefect-server

) and hitting the API (

http GET prefect-server:4200/api/health

using httpie). • I can run a python flow file like this one directly in the

prefect-agent

and it runs fine, communicating the results to the

prefect-server

. • The agent is able to spin up the flow runner container, right before it crashes. I don’t understand why the agent seems to be able to resolve the

prefect-server

hostname when running flows locally, but fails to do so when running the same flow through a deployment. Any suggestions on what may be happening, or what I can do to further debug this issue? Many thanks in advance for your help!

Hi Simon! The flow starts in a new Docker container that won't be part of the same Docker network as

prefect-agent

and

prefect-server

. However, the Docker infrastructure block has a field that lets you add names of networks to connect your flow containers to. It sounds like that might be what you need here if you haven't tried that already.

Brilliant! Setting the networks argument when creating the block worked. Thank you! I guess I have a misunderstanding on who runs what… I thought that the flow runner container didn’t need to talk to the server at all, and it was just triggered by the agent. I’m confused as why the error would show up in the agent’s logs as generated by the

prefect engine

as opposed to the

prefect.infrastructure.docker-container

reporting it… 🤔. Thank you for your help. If by chance you are aware of any good documentation or diagram that explains where the different events take place in a Prefect workflow using the Docker block, I’d appreciate if you could point me to it.