<@ULVA73B9P> What could be the reason for httpcore...
# marvin-ai
i
@Marvin What could be the reason for httpcore.ConnectError: [Errno 101] Network is unreachable on trying to submit a subflow run?
m
thought for 130.2 seconds
Most likely, that error means the process trying to create the subflow run cannot reach the Prefect API at all. In Prefect 3.x, submitting a subflow creates a flow run via the API, so if that environment cannot connect to
PREFECT_API_URL
, you can see:
Copy code
httpcore.ConnectError: [Errno 101] Network is unreachable
Common causes 1.
PREFECT_API_URL
points to the wrong place
-
localhost
/
127.0.0.1
is the big one - This works only if the API is running in the same network namespace as the flow process - If your flow is running in Docker, Kubernetes, a remote worker, or another VM,
localhost
refers to that environment, not your laptop or host machine 2. Container / pod network isolation - The worker or flow run is inside a container/pod and cannot route to the host where Prefect API is running - Example:
PREFECT_API_URL=<http://localhost:4200/api>
inside a Docker container will usually fail unless the API is also inside that same container 3. Ephemeral/local server being used in a non-local setup - If you’re relying on a local ephemeral API or a locally hosted server bound to
127.0.0.1
, remote workers/subflows won’t be able to reach it 4. DNS / hostname issue - The hostname in
PREFECT_API_URL
doesn’t resolve from the worker environment 5. Firewall / proxy / routing issue - The worker machine/container simply has no route to that host:port 6. IPv4 / IPv6 mismatch - Less common, but can happen if the API binds on IPv6 and the client tries IPv4, or vice versa What to check first From the exact environment where the parent flow is running:
Copy code
echo $PREFECT_API_URL
curl -v $PREFECT_API_URL/health
You want to confirm: - the variable is set - it includes
/api
- it is reachable from that runtime Examples - Bad in Docker/Kubernetes/remote worker:
Copy code
PREFECT_API_URL=<http://localhost:4200/api>
- Better:
Copy code
PREFECT_API_URL=<http://your-prefect-server:4200/api>
or for Prefect Cloud, the proper Cloud API URL for your workspace. If you’re running in Docker - use a reachable hostname/service name, not
localhost
- depending on setup, that might be: - Docker Compose service name -
host.docker.internal
on Mac/Windows - a host IP / cluster service DNS name Important Prefect-specific note Subflows are registered through the API, so this is usually not a “subflow bug” itself — it’s usually an API connectivity/configuration problem in the execution environment. Practical troubleshooting checklist - Verify
PREFECT_API_URL
in the worker/flow runtime - Make sure it is not
localhost
unless everything is truly local - Test
curl $PREFECT_API_URL/health
from the same container/pod/process - Confirm the API server is actually running and listening on a reachable interface - Check DNS resolution for the hostname - Check firewall/network policies/security groups Relevant docs: - Flows - Workers - Settings and profiles If you want, paste: - your
PREFECT_API_URL
- where the parent flow is running (local, Docker, k8s, EC2, etc.) - where the API is running (Cloud, local server, container, etc.) and I can help narrow down the exact reason.
i
@Marvin creating subflow runs doesn't fail every time - only some of them
m
thought for 14.7 seconds
If it only fails sometimes, that usually points away from a totally wrong
PREFECT_API_URL
and more toward an intermittent network path problem. Most likely causes in that case: 1. Transient API connectivity - brief DNS failures - short network interruptions between worker and API - flaky VPN / proxy / NAT / load balancer path - intermittent egress restrictions in the runtime environment 2. Some subflow submissions happen from different infrastructure - e.g. some parent runs land on one worker/node/container that can reach the API, while others land on one that cannot - this is especially common with Kubernetes, autoscaling VMs, mixed worker pools, or misconfigured container images 3. API service intermittently unavailable - Prefect server restarting - reverse proxy timing out - local self-hosted API under load - connection pool exhaustion / temporary socket issues 4. DNS or IPv6 intermittency - hostname resolves to multiple addresses and only some are reachable - some attempts go over IPv6 and fail, others over IPv4 succeed 5. Local/self-hosted server bound incorrectly - if the API is bound to localhost or a single interface, some execution paths may still accidentally work while others fail depending on where the flow code runs A useful mental model: - Always fails → likely bad config - Sometimes fails → likely inconsistent environment or unstable network Things I’d check next 1. Do failed runs come from a specific worker / node / container? - Compare successful vs failed parent flow runs - Check worker hostname, pod, container, node, or infrastructure identity 2. Are you self-hosting the API? - If yes, inspect API/reverse-proxy logs around failure times - Look for restarts, 5xxs, dropped connections, or resource pressure 3. What hostname is in
PREFECT_API_URL
?
- If it’s a hostname, test repeated resolution/connectivity from the runtime:
Copy code
for i in $(seq 1 20); do
  getent hosts your-api-host
  curl -sS $PREFECT_API_URL/health || echo failed
done
4. Check whether failures correlate with concurrency/load - do they happen when many subflows are being created at once? - if yes, it could be API saturation, connection exhaustion, or network device limits 5. Check for mixed environments - some workers using one profile/env var - other workers using another - some images missing network config / certs / DNS setup The strongest suspicion, given your symptom If only some subflow run submissions fail, I’d first suspect: - multiple workers/nodes with inconsistent network access, or - intermittent DNS/network issues to the API endpoint If you share: - Prefect Cloud or self-hosted? - worker type (process, docker, kubernetes, etc.) - whether failures cluster on specific workers/nodes - the exact
PREFECT_API_URL
shape (you can redact domain) I can help pinpoint the most likely root cause.
i
@Marvin it is on prefect cloud, process worker. I'm not willing to share another info
m
thought for 11.5 seconds
Got it — with Prefect Cloud + process worker + intermittent
httpcore.ConnectError: [Errno 101] Network is unreachable
, the most likely causes are: Most likely - intermittent outbound network issues on the machine running the process worker - temporary DNS failures - flaky VPN/corporate proxy - firewall/egress policy - host-level routing drops - some worker hosts can reach Prefect Cloud and some cannot - if you have multiple process workers on different machines - IPv6/path selection weirdness - some connection attempts choose an unreachable route, others succeed Less likely - bad Prefect config, because that would usually fail consistently - a subflow-specific bug, because subflow creation is just another API call Since you don’t want to share more details, here’s the shortest practical guidance: Things to verify on the worker host 1. Repeatedly test connectivity to Prefect Cloud from the same machine that runs the worker
Copy code
for i in $(seq 1 20); do
  date
  curl -I <https://api.prefect.cloud>
  sleep 2
done
2. Check whether the worker host has unstable DNS
Copy code
for i in $(seq 1 20); do
  date
  nslookup api.prefect.cloud
  sleep 2
done
3. If you use a VPN, proxy, or corporate network controls, test without them if possible. 4. Check whether failures line up with host network events: - VPN reconnects - interface resets - DNS resolver restarts - machine sleep/wake - transient packet loss Why this fits your symptom
Errno 101
is a host/network routing error. On Prefect Cloud, if only some submissions fail, that strongly suggests the worker machine occasionally cannot establish outbound connectivity to Cloud. What you can do to reduce impact - run the worker on a more stable host/network - avoid flaky VPN/proxy paths - make sure the host has consistent outbound HTTPS access - if possible, reduce network churn on that machine Docs: - Workers - Flows - Settings and profiles If you want, I can also give you a minimal local diagnostic checklist specifically for Prefect Cloud + process worker that doesn’t require sharing any environment details.