Ravish Mallya
08/22/2025, 5:05 PMwebsockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 401
when i changed the value of container environment variable PREFECT_API_URL in ecs worker from dns to private ip over httpMarvin
08/22/2025, 5:05 PMMarvin
08/22/2025, 5:07 PMPREFECT_API_URL
from a DNS name to a private IP over http can trigger that for a few common reasons:
Most likely causes
- Prefect Cloud URL or auth used with an IP: If you’re pointing a worker at Prefect Cloud, you must use the Cloud HTTPS URL and a valid PREFECT_API_KEY
. Connecting via a raw IP (and/or http) is not supported and will result in auth failures on the WS upgrade.
- Auth handled by your proxy/ingress: If you’re self-hosting behind an ALB/ingress or oauth proxy, switching to a private IP bypasses host-based routing and/or the auth layer (or breaks Host/SNI), so the backend returns 401. Many setups rely on the original DNS host for routing and token/cookie validation.
- Missing/invalid API key: If you’re using Prefect Cloud, the worker needs PREFECT_API_KEY
set and valid for the org/workspace that PREFECT_API_URL
targets.
Quick checks
1) Confirm your target:
- Prefect Cloud: Keep PREFECT_API_URL
as the Cloud URL (https) and set PREFECT_API_KEY
. Do not switch to a private IP.
- Self-hosted Prefect Server: Typically no API key is required. Use your DNS name (recommended) or the service URL your ingress exposes. If you need to use a private IP, ensure the proxy/ingress will accept that Host and forward auth if any.
2) Verify what the worker is actually using inside the ECS task:
prefect config view
echo "$PREFECT_API_URL"
echo "$PREFECT_API_KEY" | wc -c
Also enable debug to see the exact WS URL attempts:
PREFECT_LOGGING_LEVEL=DEBUG
3) Test basic connectivity from the task container (HTTP not WS) to confirm you’re hitting the right endpoint:
curl -i "$PREFECT_API_URL/health"
If this returns 401 via the private IP but 200 via DNS, you’re likely bypassing required auth or breaking host-based routing.
4) If behind an AWS ALB/Nginx/Ingress with auth:
- Stick with the DNS name so Host routing and auth work.
- Or adjust your proxy so requests to the IP still map to the correct virtual host and preserve required headers/cookies.
If you are on Prefect Cloud
- Ensure PREFECT_API_URL
is the Cloud API URL (https), not an IP.
- Ensure PREFECT_API_KEY
is set and valid for your workspace. See API keys.
If you are self-hosting
- Do not set PREFECT_API_KEY
unless you’ve intentionally added an auth layer that expects it.
- If you’re using Helm/Ingress, confirm your ingress rules and hostnames. Sticking with DNS is typically the most reliable approach. See:
- Server Helm
- Security settings
Can you share:
- Whether this is Prefect Cloud or self-hosted?
- The previous working PREFECT_API_URL
(DNS form) and the new IP form you tried (scrub any secrets)?
- Whether PREFECT_API_KEY
is set in the ECS task?
- Any proxy/ingress in front of your API (ALB, Nginx, oauth2-proxy, etc.)?
With those details I can give you exact next steps.Ravish Mallya
08/22/2025, 5:13 PMMarvin
08/22/2025, 5:13 PMMarvin
08/22/2025, 5:14 PMPREFECT_API_AUTH_STRING
, a 401 on the WebSocket after switching PREFECT_API_URL
to a private IP usually means the auth layer or routing you had on the DNS endpoint isn’t being applied to the IP endpoint. In other words, your worker is now bypassing or mismatching the proxy/vhost that enforces/forwards Basic Auth — so the WS upgrade arrives without acceptable auth and gets 401.
Why this happens when switching to an IP
- Many setups put Basic Auth on an ingress/proxy (ALB, Nginx, Traefik, etc.) that’s keyed to the DNS host. When the worker uses an IP:
- Host-based routing/auth rules may no longer match.
- You may be hitting a different server block (default vhost) that challenges or rejects.
- Some proxies strip/omit Authorization headers on certain routes unless configured.
- WebSockets must receive Authorization on the initial upgrade request; there is no challenge/response flow. Prefect will send the Authorization
header if PREFECT_API_AUTH_STRING
is set, but your proxy must accept it for both HTTP and WS paths.
Recommended fixes
- Easiest: keep using a DNS name for PREFECT_API_URL
and point it to your private IP via an internal DNS record. This preserves host-based rules and auth while staying private.
- If you must use a private IP:
- Point PREFECT_API_URL
to the proxy/ingress’s private IP (the component that applies Basic Auth), not directly to the Prefect API service.
- Update your proxy to accept requests addressed by IP (Host header = IP) and apply the same auth/routing as your DNS vhost.
Quick validations
- From the ECS worker container, verify config and that auth is present:
prefect config view
echo "$PREFECT_API_URL"
echo "$PREFECT_API_AUTH_STRING"
- Test REST with the Authorization header to both DNS and IP:
# DNS (working)
curl -i -H "Authorization: $PREFECT_API_AUTH_STRING" http://<dns-name>/api/health
# Private IP (failing)
curl -i -H "Authorization: $PREFECT_API_AUTH_STRING" http://<private-ip>/api/health
If DNS returns 200 and IP returns 401, you’re hitting a different route/vhost without the same auth rules.
- Enable debug logs to see the exact WS URL and headers sent:
PREFECT_LOGGING_LEVEL=DEBUG
Proxy-specific notes
- Nginx:
- Ensure both REST and WS locations pass auth and upgrade headers:
proxy_set_header Authorization $http_authorization;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
- Make sure the server block that handles your DNS host also handles the IP (or create a matching server block for the IP).
- AWS ALB / auth at the edge:
- You must go through the ALB hostname so auth is applied. Direct IP to a target will bypass auth and fail. Use DNS.
Docs for reference
- Security and deployment considerations: Security settings
- Helm/ingress guidance (even if not using Helm, the routing/auth patterns apply): Server Helm
If you can share:
- What component is enforcing Basic Auth (Nginx config, ALB rules, etc.)
- Whether the DNS URL worked with the exact same env vars
- A redacted snippet of your proxy config for the Prefect API paths (including any WS locations)
…I can suggest a precise change to make the IP path work, or confirm the best approach is a private DNS name for PREFECT_API_URL
.