https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I'm having trouble with provisioning...
# ask-marvin
c
@Marvin I'm having trouble with provisioning Docker container with my Prefect server and docker work pool. The container starts but then immediately crashes with an authentication error. Prefect server and docker containers are running locally. What do I need to do to fix this? Here's the log messages: 154551.655 | INFO | prefect.flow_runs.worker - Worker 'DockerWorker e4ec4945-358e-47de-b99f-9791f23fcfe0' submitting flow run '73572e8b-9eb2-4fc7-b7ac-5e64bdbf8f5f' 154551.754 | INFO | prefect.workers.docker.dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Creating Docker container 'succinct-mosquito'... 154551.754 | INFO | prefect.workers.docker.dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Creating Docker container 'succinct-mosquito'... 154551.783 | INFO | prefect.workers.docker.dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Docker container 'succinct-mosquito' has status 'created' 154551.783 | INFO | prefect.workers.docker.dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Docker container 'succinct-mosquito' created successfully. 154551.885 | INFO | prefect.workers.docker.dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Docker container 'succinct-mosquito' has status 'running' 154551.888 | INFO | prefect.flow_runs.worker - Completed submission of flow run '73572e8b-9eb2-4fc7-b7ac-5e64bdbf8f5f' 044552.775 | DEBUG | prefect.profiles - Using profile 'ephemeral' 044552.949 | DEBUG | prefect.runner - Starting runner... 044552.957 | DEBUG | prefect.client - Connecting to API at http://127.0.0.1:4200/api/ 044552.957 | DEBUG | prefect.events.clients - Reconnecting websocket connection. 044552.958 | DEBUG | prefect.events.clients - Opening websocket connection. 044552.960 | DEBUG | prefect.events.clients - Pinging to ensure websocket connected. 044552.961 | DEBUG | prefect.events.clients - Pong received. Websocket connected. 044552.961 | DEBUG | prefect.events.clients - Authenticating... 044552.971 | WARNING | prefect.events.clients - Unable to connect to 'ws://127.0.0.1:4200/api/events/in'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: Unable to authenticate to the event stream. Please ensure the provided auth_token you are using is valid for this environment. . Set PREFECT_DEBUG_MODE=1 to see the full error. Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/prefect/events/clients.py", line 374, in _auth_handshake message: Dict[str, Any] = orjson.loads(await self._websocket.recv()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/websockets/asyncio/connection.py", line 324, in recv raise self.protocol.close_exc from self.recv_exc websockets.exceptions.ConnectionClosedError: no close frame received or sent The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/prefect/cli/_utilities.py", line 44, in wrapper return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/cli/_types.py", line 159, in sync_fn return asyncio.run(async_fn(*args, **kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/cli/flow_run.py", line 676, in execute await runner.execute_flow_run(id) File "/usr/local/lib/python3.11/site-packages/prefect/runner/runner.py", line 682, in execute_flow_run async with context: File "/usr/local/lib/python3.11/site-packages/prefect/runner/runner.py", line 1682, in aenter await self._exit_stack.enter_async_context(self._events_client) File "/usr/local/lib/python3.11/contextlib.py", line 650, in enter_async_context result = await _enter(cm) ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/prefect/events/clients.py", line 305, in aenter await self._reconnect() File "/usr/local/lib/python3.11/site-packages/prefect/events/clients.py", line 417, in _reconnect await self._auth_handshake() File "/usr/local/lib/python3.11/site-packages/prefect/events/clients.py", line 388, in _auth_handshake raise Exception(msg) from e Exception: Unable to authenticate to the event stream. Please ensure the provided auth_token you are using is valid for this environment. An exception occurred. 154553.159 | INFO | prefect.workers.docker.dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Docker container 'succinct-mosquito' has status 'removing' 154553.162 | WARNING | prefect.workers.docker.dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Docker container succinct-mosquito was removed before we could wait for its completion. 154553.181 | INFO | prefect.flow_runs.worker - Reported flow run '73572e8b-9eb2-4fc7-b7ac-5e64bdbf8f5f' as crashed: Flow run infrastructure exited with non-zero status code 1.
m
Your message was too long, here's your message at the allowed limit: I'm having trouble with provisioning Docker container with my Prefect server and docker work pool The container starts but then immediately crashes with an authentication error Prefect server and docker containers are running locally What do I need to do to fix this? Here's the log messages 15 45 51 655 | INFO | prefect flow_runs worker - Worker 'DockerWorker e4ec4945-358e-47de-b99f-9791f23fcfe0' submitting flow run '73572e8b-9eb2-4fc7-b7ac-5e64bdbf8f5f' 15 45 51 754 | INFO | prefect workers docker dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Creating Docker container 'succinct-mosquito' 15 45 51 754 | INFO | prefect workers docker dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Creating Docker container 'succinct-mosquito' 15 45 51 783 | INFO | prefect workers docker dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Docker container 'succinct-mosquito' has status 'created' 15 45 51 783 | INFO | prefect workers docker dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Docker container 'succinct-mosquito' created successfully 15 45 51 885 | INFO | prefect workers docker dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Docker container 'succinct-mosquito' has status 'running' 15 45 51 888 | INFO | prefect flow_runs worker - Completed submission of flow run '73572e8b-9eb2-4fc7-b7ac-5e64bdbf8f5f' 04 45 52 775 | DEBUG | prefect profiles - Using profile 'ephemeral' 04 45 52 949 | DEBUG | prefect runner - Starting runner 04 45 52 957 | DEBUG | prefect client - Connecting to API at <http //127 0 0 1 4200/api/> 04 45 52 957 | DEBUG | prefect events clients - Reconnecting websocket connection 04 45 52 958 | DEBUG | prefect events clients - Opening websocket connection 04 45 52 960 | DEBUG | prefect events clients - Pinging to ensure websocket connected 04 45 52 961 | DEBUG | prefect events clients - Pong received Websocket connected 04 45 52 961 | DEBUG | prefect events clients - Authenticating 04 45 52 971 | WARNING | prefect events clients - Unable to connect to '<ws //127 0 0 1 4200/api/events/in>' Please check your network settings to ensure websocket connections to the API are allowed Otherwise event data (including task run data) may be lost Reason Unable to authenticate to the event stream Please ensure the provided auth_token you are using is valid for this environment Set PREFECT_DEBUG_MODE=1 to see the full error Traceback (most recent call last) File /usr/local/lib/python3 11/site-packages/prefect/events/clients py line 374 in _auth_handshake message Dict[str Any] = orjson loads(await self _websocket recv()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3 11/site-packages/websockets/asyncio/connection py line 324 in recv raise self protocol close_exc from self recv_exc websockets exceptions ConnectionClosedError no close frame received or sent The above exception was the direct cause of the following exception Traceback (most recent call last) File /usr/local/lib/python3 11/site-packages/prefect/cli/_utilities py line 44 in wrapper return fn(*args *kwargs) ^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3 11/site-packages/prefect/cli/_types py line 159 in sync_fn return asyncio run(async_fn(*args *kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3 11/asyncio/runners py line 190 in run return runner run(main) ^^^^^^^^^^^^^^^^ File /usr/local/lib/python3 11/asyncio/runners py line 118 in run return self _loop run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3 11/asyncio/base_events py line 654 in run_until_complete return future result() ^^^^^^^^^^^^^^^ File /usr/local/lib/python3 11/site-packages/prefect/cli/flow_run py line 676 in execute await runner execute_flow_run(id) File /usr/local/lib/python3 11/site-packages/prefect/runner/runner
c
@Marvin I'm having trouble with provisioning Docker container with my Prefect server and docker work pool. The container starts but then immediately crashes with an authentication error. Prefect server and docker containers are running locally. What do I need to do to fix this? Here's the log messages: 154551.655 | INFO | prefect.flow_runs.worker - Worker 'DockerWorker e4ec4945-358e-47de-b99f-9791f23fcfe0' submitting flow run '73572e8b-9eb2-4fc7-b7ac-5e64bdbf8f5f' 154551.754 | INFO | prefect.workers.docker.dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Creating Docker container 'succinct-mosquito'... 154551.754 | INFO | prefect.workers.docker.dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Creating Docker container 'succinct-mosquito'... 154551.783 | INFO | prefect.workers.docker.dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Docker container 'succinct-mosquito' has status 'created' 154551.783 | INFO | prefect.workers.docker.dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Docker container 'succinct-mosquito' created successfully. 154551.885 | INFO | prefect.workers.docker.dockerworker e4ec4945-358e-47de-b99f-9791f23fcfe0 - Docker container 'succinct-mosquito' has status 'running' 154551.888 | INFO | prefect.flow_runs.worker - Completed submission of flow run '73572e8b-9eb2-4fc7-b7ac-5e64bdbf8f5f' 044552.775 | DEBUG | prefect.profiles - Using profile 'ephemeral' 044552.949 | DEBUG | prefect.runner - Starting runner... 044552.957 | DEBUG | prefect.client - Connecting to API at http://127.0.0.1:4200/api/ 044552.957 | DEBUG | prefect.events.clients - Reconnecting websocket connection. 044552.958 | DEBUG | prefect.events.clients - Opening websocket connection. 044552.960 | DEBUG | prefect.events.clients - Pinging to ensure websocket connected. 044552.961 | DEBUG | prefect.events.clients - Pong received. Websocket connected. 044552.961 | DEBUG | prefect.events.clients - Authenticating... 044552.971 | WARNING | prefect.events.clients - Unable to connect to 'ws://127.0.0.1:4200/api/events/in'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: Unable to authenticate to the event stream. Please ensure the provided auth_token you are using is valid for this environment. . Set PREFECT_DEBUG_MODE=1 to see the full error. Exception: Unable to authenticate to the event stream. Please ensure the provided auth_token you are using is valid for this environment. An exception occurred.
m
✅ thought for 348.0 seconds
Thanks for the logs — they point to a specific issue. What’s happening - Your job container connects to the Prefect Server event websocket, then fails at the auth handshake: - “Authenticating…” followed by “Unable to authenticate to the event stream…” - In Prefect 3.x self‑hosted, the event stream requires a token if the server was started with 
PREFECT_SERVER_API_AUTH_STRING
. Any client (workers, job containers, scripts) must send the matching 
PREFECT_API_AUTH_STRING
. You likely have one (or both) of these issues: 1) Token mismatch/missing - Server has 
PREFECT_SERVER_API_AUTH_STRING
set, but the job container does not have the matching 
PREFECT_API_AUTH_STRING
. 2) Wrong API URL inside the container - Inside the job container, 
PREFECT_API_URL
is defaulting to 
<http://127.0.0.1:4200/api>
(your logs show that), which is usually not the host server when running in Docker. Even if auth is fixed, the container should point at a reachable host address. How to fix Pick one of the two approaches below: A) Keep auth enabled (recommended if you set it intentionally) 1. Confirm the server is started with an auth string: - If you’re starting locally:
Copy code
PREFECT_SERVER_API_AUTH_STRING="admin:password" prefect server start
2. Propagate the same auth string and a correct API URL to your job containers via the Docker work pool’s base job template (UI: Work Pools > your docker pool > Base job template > Edit): - Add environment:
Copy code
"env": {
       "PREFECT_API_URL": "<http://host.docker.internal:4200/api>",   // Mac/Windows
       "PREFECT_API_AUTH_STRING": "admin:password"
     }
- If you’re on Linux, replace the URL or add host mapping: - Use the bridge gateway IP:
Copy code
"env": {
         "PREFECT_API_URL": "<http://172.17.0.1:4200/api>",
         "PREFECT_API_AUTH_STRING": "admin:password"
       }
- Or add an extra host mapping equivalent to 
--add-host=host.docker.internal:host-gateway
in the base job template and keep 
host.docker.internal
in the URL. 3. (Optional) Verify the pool config via CLI
Copy code
prefect work-pool inspect "your-docker-pool" --output json
If needed, export, edit, and re-apply the template:
Copy code
# edit docker.json to add env settings above
   prefect work-pool update "your-docker-pool" --base-job-template docker.json
B) Disable auth (if you didn’t mean to enable it for local dev) - Unset 
PREFECT_SERVER_API_AUTH_STRING
and restart the server. Then you won’t need 
PREFECT_API_AUTH_STRING
in the container. - You still should set a correct 
PREFECT_API_URL
inside the job container (see above) — don’t leave it at 
<http://127.0.0.1:4200/api>
unless you’re using host networking. Notes on API URL inside Docker - Mac/Windows (Docker Desktop): 
<http://host.docker.internal:4200/api>
- Linux (default bridge): 
<http://172.17.0.1:4200/api>
OR add a host-gateway mapping and use 
host.docker.internal
- If your Server is also running in Docker Compose on the same network, use the service name: e.g., 
<http://prefect-server:4200/api>
Why your container crashed - The runner raises an exception when the auth handshake fails with the events service. Once you provide a valid 
PREFECT_API_AUTH_STRING
(matching the server’s 
PREFECT_SERVER_API_AUTH_STRING
) and a reachable 
PREFECT_API_URL
, the flow should start and report events normally. Docs and references - Manage settings (including 
PREFECT_API_URL
): Manage settings - Security settings (Basic auth for self‑hosted: 
PREFECT_SERVER_API_AUTH_STRING
and 
PREFECT_API_AUTH_STRING
): Security settings - Run Prefect Server in Docker: Server in Docker
- Run flows in Docker containers (work pools/workers): Run flows in Docker containers Quick questions to confirm so I can tailor exact snippets: - Are you on macOS/Windows or Linux? - Did you intentionally set 
PREFECT_SERVER_API_AUTH_STRING
when starting the server? If so, what’s the value format (no need to share the secret itself)? - Is your Server running on the host machine or in Docker Compose? If Compose, what’s the service name/network setup?
c
@Marvin PREFECT_SERVER_API_AUTH_STRING is set to 'None' and comes from the defaults. how can I unset ?
m
✅ thought for 86.4 seconds
If you literally have the string "None" set, Prefect will treat that as a real auth string and require clients to send "None" — which causes the websocket auth failure you’re seeing. You’ll want to clear it in two places, depending on how it was set: 1) Unset the environment variable (this is what the server actually reads) - Bash/zsh:
Copy code
unset PREFECT_SERVER_API_AUTH_STRING
- PowerShell:
Copy code
Remove-Item Env:PREFECT_SERVER_API_AUTH_STRING
- Windows CMD:
Copy code
set PREFECT_SERVER_API_AUTH_STRING=
If you’re using Docker Compose, remove it from the service’s environment block (or set it to an empty string, which disables auth because it’s falsy):
Copy code
services:
  prefect-server:
    environment:
      # Remove this line entirely, or make it empty
      # PREFECT_SERVER_API_AUTH_STRING:
Then restart the server container. 2) If you set it in a Prefect profile, unset it there too - This restores the default (None) in the active profile:
Copy code
prefect config unset PREFECT_SERVER_API_AUTH_STRING
Verify it’s gone - Check the shell you’ll use to start the server: - Mac/Linux:
Copy code
echo $PREFECT_SERVER_API_AUTH_STRING
- PowerShell:
Copy code
$Env:PREFECT_SERVER_API_AUTH_STRING
- And view Prefect’s profile settings:
Copy code
prefect config view
Important: Restart the Prefect server after unsetting so it picks up the change. Tip: If your job containers are still pointing at 127.0.0.1 from inside Docker, update 
PREFECT_API_URL
too: - Mac/Windows: 
<http://host.docker.internal:4200/api>
- Linux: 
<http://172.17.0.1:4200/api>
(or add a host-gateway mapping and use host.docker.internal) Docs for reference: - Security settings (auth strings): Security settings - Manage settings and profiles: Manage settings If you clear that env var and restart the server, the websocket should accept connections without a token and your container should stop crashing.
3 Views
Open in Slack
PreviousNext
Powered by