Great question — the short version: 127.0.0.1 inside a Docker container refers to that container itself, not your host or other containers. To let your Prefect flow-run containers reach your FastAPI, you have three solid options: Option A: Reach the host via host.docker.internal - Use http://host.docker.internal:1000 in your flow code. - On Linux, add the host-gateway mapping to the flow-run containers so host.docker.internal resolves. - Configure this in your Docker work pool or per deployment. Prefect 3.x deployment-level override (prefect.yaml):

deployments:
  - name: my-deployment
    work_pool_name: my-docker-pool
    job_variables:
      container_create_kwargs:
        extra_hosts:
          host.docker.internal: host-gateway

Or via Python when deploying:

from prefect import flow

@flow
def my_flow():
    import requests
    r = requests.get("<http://host.docker.internal:1000/health>")
    print(r.text)

if __name__ == "__main__":
    my_flow.deploy(
        name="my-deployment",
        work_pool_name="my-docker-pool",
        job_variables={
            "container_create_kwargs": {
                "extra_hosts": {"host.docker.internal": "host-gateway"}
            }
        },
    )

Option B: Put your flow containers on the same Docker network as your FastAPI container - Attach your FastAPI container to a user-defined network (e.g., my-net). - Configure the Prefect Docker work pool or deployment to join that same network. - Call the API by its container or service name and port (e.g., http://my-fastapi:1000). - Ensure your FastAPI is bound to 0.0.0.0 in its container. Deployment-level override:

deployments:
  - name: my-deployment
    work_pool_name: my-docker-pool
    job_variables:
      networks:
        - my-net

Then in your flow:

import requests
requests.get("<http://my-fastapi:1000/health>")

Notes: - The network (my-net) must already exist on the Docker host, and your FastAPI container must be attached to it. - Using a user-defined network with service-name DNS is usually the most reliable for container-to-container calls. Option C: Use host networking for flow runs (Linux only) - Set network_mode: host so the container shares the host network namespace and can hit 127.0.0.1:1000 directly. - Caveats: This is Linux-only, disables port mapping, and is generally less portable. Deployment-level override:

deployments:
  - name: my-deployment
    work_pool_name: my-docker-pool
    job_variables:
      network_mode: host

Which should you choose? - If the API is running in another container: Option B (same network + service name) is best. - If it’s published on your host at 127.0.0.1:1000 and you want to keep it that way: Option A with host.docker.internal is simplest and portable. On Linux, include the extra_hosts mapping above. - If you’re on Linux and want the simplest “it just works with localhost”: Option C (network_mode=host). Helpful docs: - Docker worker how-to - Customize job variables - Prefect Docker integration Quick question to confirm the best path: Is your FastAPI app running as a separate Docker container, or is it bound on your host and published to 127.0.0.1:1000 from there?

Short answer: Yes — on Windows with Docker Desktop, Option A works out of the box. Details: - On Windows and macOS, containers can reach services running on your host via http://host.docker.internal. No extra

extra_hosts

mapping is needed. - If your FastAPI app is running in another Docker project/container, Option A will still work as long as that app is published to the host (e.g.,

-p 1000:1000

or a Compose

ports:

entry). Your flow-run containers will call the published host port via

host.docker.internal:1000

. So you can do this in your flow:

import requests

def call_api():
    r = requests.get("<http://host.docker.internal:1000/health>")
    print(r.text)

When to use Option B instead: - If your FastAPI container is not publishing its port to the host, or you prefer container-to-container networking, attach your Prefect flow-run containers to the same user-defined Docker network and call the API by its service/container name:

# deployment job variables
job_variables:
  networks: ["my-net"]

Then in code:

requests.get("<http://my-fastapi:1000/health>")

Make sure your FastAPI container is on

my-net

and listening on

0.0.0.0

. Note: Option C (network_mode=host) is not supported on Windows, so stick with A or B. Quick check: Can you confirm your FastAPI service is reachable from your host browser at http://127.0.0.1:1000? If yes, Option A will work immediately for your Prefect flow-run containers.

Yes — on Windows with Docker Desktop, your Prefect flow-run containers can reach services exposed on the host using host.docker.internal. That includes your Redpanda broker’s “external” ports. Key points to make it work end-to-end: - Use the host-published external port(s) in your client config inside the flow container, e.g. bootstrap servers = host.docker.internal:19092 (adjust to your external port). - Make sure Redpanda’s advertised address for the external listener matches how your clients connect. If Redpanda advertises localhost:19092 but your flow runs in a container, the client will get redirected to localhost:19092 (which would be the container itself) and fail. Typical Redpanda Docker config with dual listeners - Internal listener for other containers on the same Docker network - External listener published to the host and advertised as host.docker.internal for containerized clients Example docker-compose for Redpanda:

services:
  redpanda:
    image: redpandadata/redpanda:latest
    container_name: redpanda
    ports:
      - "19092:19092"   # external listener published to host
    command:
      - redpanda start
      - --overprovisioned
      - --smp
      - "1"
      - --memory
      - 1G
      - --reserve-memory
      - 0M
      - --check=false
      - --node-id
      - "0"
      - --kafka-addr
      - <PLAINTEXT://0.0.0.0:9092,OUTSIDE://0.0.0.0:19092>
      - --advertise-kafka-addr
      - <PLAINTEXT://redpanda:9092,OUTSIDE://host.docker.internal:19092>
    networks:
      - my-net

networks:
  my-net:
    driver: bridge

Then in your Prefect flow (runs in a Docker container):

from confluent_kafka import Producer

p = Producer({"bootstrap.servers": "host.docker.internal:19092"})
p.produce("my-topic", b"hello")
p.flush()

Alternative: same Docker network - If you put your Prefect flow-run containers and Redpanda on the same user-defined Docker network, you can use the internal listener (e.g., redpanda:9092) and avoid host.docker.internal entirely. - For Prefect, set your Docker work pool/deployment job_variables to join that network:

deployments:
  - name: my-deployment
    work_pool_name: my-docker-pool
    job_variables:
      networks: ["my-net"]

Then use bootstrap servers redpanda:9092 in your flow. Summary: - On Windows, host.docker.internal works from containers to host-published ports, so yes, your flow containers can reach Redpanda’s external ports that way. - Ensure Redpanda’s advertised addresses for the external listener are host.docker.internal:<port>, otherwise clients will get unusable broker addresses. If you also have non-container clients on the host that need different addresses, consider: - Using a DNS name that resolves appropriately in both environments, or - Providing two listeners: one internal (redpanda:9092) and one external (host.docker.internal:19092), and point each client type at the right one. If you share your exact Redpanda port mappings and current advertised listener settings, I can suggest the precise values to use.

Given your setup, the cleanest and most reliable approach is to put all Prefect flow-run containers on the same Docker network as your FastAPI and Redpanda containers, then use service/container names for connectivity. Since you’ll use both across flows, setting this at the Docker work pool level is ideal so every run inherits it automatically. Why this is best for your case - Your Redpanda compose already defines internal and external listeners. Prefect runs can use the internal listener via DNS on the shared network (avoids advertised-address pitfalls). - Your FastAPI is already on that network; calling it by service/container name avoids needing host.docker.internal. - You only configure the network once (in the work pool), not per deployment. What to configure 1) Add your compose network to the Docker work pool’s base job template - In the Prefect UI: Work Pools -> your Docker pool -> Edit Base Job Template -> under job_configuration add:

"networks": ["redpanda_network"]

Note: Use the actual network name as Docker sees it (check with

docker network ls

). If your compose declares a custom network name (e.g., name: redpanda-local), that’s the string you should use. - Or via CLI: - Inspect and save the template:

prefect work-pool inspect "my-docker-pool" > base.json

- Edit base.json and add the network under job_configuration:

{
      "job_configuration": {
        "networks": ["redpanda_network"]
        // ... keep your other settings as-is
      }
      // ... rest of template
    }

- Update the pool:

prefect work-pool update "my-docker-pool" --base-job-template base.json

2) Update your flow code to use service names - FastAPI (replace with your actual service/container name):

import requests
requests.get("<http://fastapi-service-name:1000/health>")

- Redpanda internal listener:

from confluent_kafka import Producer

p = Producer({"bootstrap.servers": "redpanda-0:9092"})
p.produce("my-topic", b"hello")
p.flush()

A note on host.docker.internal - On Windows, containers can reach host-published ports via host.docker.internal, so Option A does work. - However, for Redpanda specifically, using its external listener from containers requires the external advertised address to match what the client can resolve. Your compose currently sets: - --advertise-kafka-addr ... external://localhost:19092 - That’s perfect for host-based clients, but not for containers (they’ll get “localhost” back and fail). - You could change that to external://host.docker.internal:19092 to support container clients, but then host clients may not resolve host.docker.internal without extra steps. This is why the shared network approach is simpler: containers use internal listener redpanda-0:9092; host clients keep using localhost:19092. If you prefer per-deployment instead of pool-wide - Add job variables in prefect.yaml:

deployments:
  - name: my-deployment
    work_pool_name: my-docker-pool
    job_variables:
      networks: ["redpanda_network"]

Docs - Docker worker how-to - Customize job variables Quick sanity checks - What is the exact network name from

docker network ls

that your compose created (e.g., redpanda-local vs redpanda_network)? Use that exact string in the work pool. - What is the FastAPI service/container name you want flows to call? I can give you the exact URL to use once I know it.

Great questions. Short answers first: - Yes, your Prefect flow-run containers can join a Docker network created by a different docker-compose project. - You do not need to change your Prefect compose just for flow-run containers to use that network. - Use the exact network name reported by docker (in your case: redpanda-local_redpanda_network). Details 1) Cross-compose networking - Docker networks are host-level objects. Any container (no matter which compose file created it) can join an existing user-defined bridge network by name. - Configure your Prefect Docker work pool (or per-deployment job variables) to attach flow-run containers to that network:

# Option A: pool-wide (Base Job Template)
{
  "job_configuration": {
    "networks": ["redpanda-local_redpanda_network"]
  }
}

Update via:

prefect work-pool inspect "my-docker-pool" > base.json
# edit base.json to add the networks entry above
prefect work-pool update "my-docker-pool" --base-job-template base.json

Or per deployment in prefect.yaml:

deployments:
  - name: my-deployment
    work_pool_name: my-docker-pool
    job_variables:
      networks: ["redpanda-local_redpanda_network"]

Then your flows can use: - FastAPI: http//&lt;fastapi service name&gt;1000 - Redpanda internal listener: bootstrap servers = redpanda-0:9092 2) Do Prefect services need the network? - Not for launching flow runs. The Docker worker will create flow-run containers and attach them to the network even if the worker or Prefect API/UI are not on that network. - Only add this network to your Prefect compose if Prefect services themselves need to talk to FastAPI or Redpanda. If so, declare the network as external in your Prefect compose:

networks:
  redpanda_network:
    external: true
    name: redpanda-local_redpanda_network

services:
  prefect-worker:
    # ...
    networks:
      - redpanda_network

3) About the network name - Since docker network ls shows redpanda-local_redpanda_network, use that exact string in Prefect job variables. - Optional: If you want a stable, shorter name without the compose project prefix, set an explicit network name in your Redpanda compose:

networks:
  redpanda_network:
    name: redpanda_network
    driver: bridge

Then re-create the stack; you’d reference networks: ["redpanda_network"] from Prefect. Reminders - For Redpanda, prefer the internal listener inside the shared network: redpanda-0:9092. Leave your external listener (localhost:19092) for host apps. - For FastAPI, ensure it listens on 0.0.0.0 in its container; call it by the service/container name on the shared network. Docs - Docker worker how-to - Customize job variables - Manage work pools If you share the FastAPI service name from its compose, I can give you the exact URL to use in your flows.

Short answer: Yes — putting the network in the base job template’s job_configuration is sufficient. You only need to add it to the variables section if you want it to be configurable per-deployment or editable via the UI. What that means in practice - Hardcode (simple, applies to all runs; not overrideable by deployments):

{
  "job_configuration": {
    "networks": ["redpanda-local_redpanda_network"]
    // ... other settings
  }
  // variables section can stay as-is
}

- Expose as a variable (lets you override per deployment with job_variables or change it in the UI):

{
  "job_configuration": {
    "networks": "{{ networks }}"
  },
  "variables": {
    "type": "object",
    "properties": {
      "networks": {
        "title": "Networks",
        "type": "array",
        "items": {
          "type": "string"
        },
        "default": [
          "redpanda-local_redpanda_network"
        ]
      }
    }
  }
}

Then a deployment can override:

job_variables:
  networks: ["some-other-network"]

Given your goal (all flows should reach FastAPI and Redpanda), I’d recommend hardcoding the network in job_configuration at the work pool level using the exact name Docker shows: redpanda-local_redpanda_network. That way every flow-run container automatically joins that network and can use: - FastAPI: http//&lt;your fastapi service name&gt;1000 - Redpanda: bootstrap.servers=redpanda-0:9092 CLI helper (optional):

prefect work-pool inspect "my-docker-pool" > base.json
# edit base.json to add the networks entry as shown above
prefect work-pool update "my-docker-pool" --base-job-template base.json

Docs: - Docker worker how-to - Customize job variables - Manage work pools

- Redpanda address: use just

redpanda-0:9092

as the value. The

bootstrap.servers

part is the config key, not part of the address string. - confluent_kafka:

from confluent_kafka import Producer
    p = Producer({"bootstrap.servers": "redpanda-0:9092"})

- kafka-python:

from kafka import KafkaProducer
    p = KafkaProducer(bootstrap_servers="redpanda-0:9092")

- aiokafka:

from aiokafka import AIOKafkaProducer
    p = AIOKafkaProducer(bootstrap_servers="redpanda-0:9092")

- FastAPI hostname inside the shared network: - If your Compose service is named

broker_service

, other containers on the same user-defined network can typically reach it at

<http://broker_service:1000>

. - Compose also prefixes container names with the project (e.g.,

redpanda-local-broker_service-1

) unless you set

container_name

, but DNS resolution on the network generally exposes the service name as an alias, so

broker_service

should work across projects on that network. - If you want to make it explicit (or if

broker_service

doesn’t resolve from outside the same Compose project), add a network alias or set a fixed container name: - Using a network alias:

services:
        broker_service:
          networks:
            redpanda_network:
              aliases:
                - broker_service

- Or set a fixed container name:

services:
        broker_service:
          container_name: broker_service
          networks:
            - redpanda_network

Given you’ve attached your Prefect flow-run containers to

redpanda-local_redpanda_network

via the work pool, they’ll be able to reach: - Redpanda at

redpanda-0:9092

- FastAPI at

broker_service:1000

(assuming that’s the service or alias name on that network) If you share the exact FastAPI service stanza, I can confirm the best hostname to use.