https://prefect.io logo
Channels
Powered by
t

Tim-Oliver

08/04/2023, 3:21 PM
Has anyone encountered this error in the prefect-agent run output? I get usually 2 lines of expected log and then this error message. The agent runs, however the agent log is super polluted and hard to read with all the error messages.
j

Jake Kaplan

08/04/2023, 4:41 PM
Are you able to see client side events from your flow runs? like task run events? events are sent over a websocket connection and your network may be blocking it?
t

Tim-Oliver

08/07/2023, 7:48 AM
I am able to see events on the the event feed page in cloud UI. Is this what you were referring to?
j

Jake Kaplan

08/07/2023, 2:28 PM
sorry are you able to see task run events? or any other client side generated events? my hunch is your network configuration is not letting you open a websocket connection
t

Tim-Oliver

08/07/2023, 3:59 PM
I see only 
prefect.flow_run
and 
prefect.log
events in cloud UI. I am in a rather restricted environment and your hunch is probably right on point.
Is there something I could configure on the prefect side? Or what requirements would need to be fullfilled on the network config side? I don't have high hopes that the network config can be changed, but I could ask if I know the necessary requirements.
j

Jake Kaplan

08/07/2023, 4:35 PM
It's not something you're able to disable, but on the network side you'd need to be able to open a persistent websocket connection (for the duration of the flow run). It goes to 
<wss://api.prefect.cloud/.../>
(as opposed to 
https://...
) you might need to whitelist the url or ip? Unfortunately it's really really dependent on the network setup what is locked down and how it's locked down
t

Tim-Oliver

08/30/2023, 9:36 AM
Hi, I was on vacation. I checked with my IT and got the following answer:
Copy code
"By default the WebSocket protocol uses port 80 for regular WebSocket connections and port 443 for WebSocket connections tunneled over TLS [RFC2818]."
Since you need to use our proxy for outgoing http/https connections, a direct access attempt fails. If prefect does not allow configuring a proxy, you may try defining it via environment variables http_proxy and https_proxy (e.g.: https_proxy=<our-proxy>) for starting the server process.
I have set the environment variables accordingly, but the 
EventsWorker
Service is still not able to communicate. Is this service respecting the env-vars? Thank you for helping me figuring this out! Kind regards, Tim-Oliver
j

Jake Kaplan

08/31/2023, 12:54 PM
Hi Tim, proxies are supported. You should find the docs here
t

Tim-Oliver

08/31/2023, 1:48 PM
Hi Jake, This is what I did. And I guess it is working in general. Otherwise I would expect that the agent can´t connect at all to prefect-cloud. However, in my case only the 
EventsWorker
has troubles connecting. Is there something I could do to debug this further? Thanks!
Hi Jake, I just updated to the latest prefect version and I see that the agent log is getting less polluted by this issue. It is now just a one-liner: 
16:41:41.688 | ERROR  | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 1 pending items.
That already helps a lot with making the log digestible. Thank you very much! But I am still wondering how I could get this to run as it is intended. I tried to reproduce the error by opening a websocket outside of prefect:
Copy code
import asyncio
from websockets.legacy import client

c = client.Connect("<wss://34.111.36.90>", extra_headers={"Authorization": "bearer API-key"})

loop = asyncio.get_event_loop() 
task = loop.create_task(c.__aenter__())
loop.run_until_complete(asyncio.wait([task]))
With this I get 
ConnectionRefusedError: [Errno 111] Connect call failed ('34.111.36.90', 443)
which is not the same error as above. Could you point me in the right direction to get the above error reproduced with a minimal example? Any help is much appreciated!
Powered by