https://prefect.io logo
Join Slack
Powered by
can anyone help me shed some light on this warning...
# ask-community
d
can anyone help me shed some light on this warning and what might be wrong?
Copy code
13:22:08.805 | WARNING | prefect.events.clients - Unable to connect to '<wss://workflows.arte.adobe.net/api/events/in>'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: server rejected WebSocket connection: HTTP 403. Set PREFECT_DEBUG_MODE=1 to see the full error.
13:22:08.805 | ERROR   | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 11 pending items.
full output in thread
Copy code
-> python my_workflow.py
13:22:06.160 | INFO    | prefect.engine - Created flow run 'sceptical-jaguarundi' for flow 'show-stars'
13:22:06.162 | INFO    | prefect.engine - View at <https://workflows.arte.adobe.net/runs/flow-run/72983fda-177b-4d7c-8c3b-3a6be4f7b026>
13:22:07.103 | INFO    | Task run 'fetch_stats-7ac' - Finished in state Completed()
13:22:07.175 | INFO    | Task run 'get_stars-bd5' - Finished in state Completed()
13:22:07.176 | INFO    | Flow run 'sceptical-jaguarundi' - PrefectHQ/prefect: 17687 stars
13:22:07.467 | WARNING | prefect.events.clients - Unable to connect to '<wss://workflows.arte.adobe.net/api/events/in>'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: server rejected WebSocket connection: HTTP 403. Set PREFECT_DEBUG_MODE=1 to see the full error.
13:22:07.468 | ERROR   | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 8 pending items.
13:22:07.619 | INFO    | Task run 'fetch_stats-0b9' - Finished in state Completed()
13:22:07.686 | INFO    | Task run 'get_stars-816' - Finished in state Completed()
13:22:07.687 | INFO    | Flow run 'sceptical-jaguarundi' - pydantic/pydantic: 21497 stars
13:22:08.145 | INFO    | Task run 'fetch_stats-361' - Finished in state Completed()
13:22:08.208 | INFO    | Task run 'get_stars-bc2' - Finished in state Completed()
13:22:08.209 | INFO    | Flow run 'sceptical-jaguarundi' - huggingface/transformers: 135910 stars
13:22:08.257 | INFO    | Flow run 'sceptical-jaguarundi' - Finished in state Completed()
13:22:08.805 | WARNING | prefect.events.clients - Unable to connect to '<wss://workflows.arte.adobe.net/api/events/in>'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: server rejected WebSocket connection: HTTP 403. Set PREFECT_DEBUG_MODE=1 to see the full error.
13:22:08.805 | ERROR   | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 11 pending items.
the script is from the quickstart guide: https://docs.prefect.io/v3/get-started/quickstart
I'm running my prefect server in a Kubernetes instance and wondering if I need to adjust the k8s service or load balancer to allow wss connections
it's currently using the TCP protocol and I thought that would be sufficient
n
hi @Devin Flake - are you in a place that disallows outbound websockets?
132208.805 | ERROR | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 11 pending items.
this means "websocket client cant send up events for some reason"
d
that's probably the issue but I just wanted a second opinion 🙂
👍 1
n
ope yeah, looking closer
132207.467 | WARNING | prefect.events.clients - Unable to connect to 'wss://workflows.arte.adobe.net/api/events/in'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: server rejected WebSocket connection: HTTP 403. Set PREFECT_DEBUG_MODE=1 to see the full error.
👍 1
d
thanks for looking Nate!
n
catjam
d
fyi - that fixed it: 
enableWebsockets: true
Copy code
apiVersion: <http://contour.heptio.com/v1beta1|contour.heptio.com/v1beta1>
kind: IngressRoute
metadata:
  name: prefect-ingress-tls
  labels:
    environment: production
    app: prefect
spec:
  virtualhost:
    fqdn: <http://workflows.arte.adobe.net|workflows.arte.adobe.net>
    tls:
      secretName: arte-star-tls
  routes:
    - enableWebsockets: true
      match: /
      timeout: 900s
      permitInsecure: false
      services:
        - name: prefect-server
          port: 4200
n
thank you! would you mind commenting here to say something to that affect? (in case its helpful for anyone with a similar setup) otherwise I can copy this snippet over there
d
done - thanks again Nate!
n
thank you!
164 Views
Open in Slack
PreviousNext
Powered by