We got a 503 on the websocket connection that crashed the ECS observer about 20 minutes ago (SQS queue depth alarm just went off). @alex this is the first we've seen of this in a while (thank you for the changes made to bring stability to the observer bits). Curious if this was a one-off for our account?

21:20:06.494 | WARNING | prefect.events.clients - Unable to connect to '<wss://api.prefect.cloud/api/accounts/><id>/workspaces/<id>/events/in'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: server rejected WebSocket connection: HTTP 503. Set PREFECT_DEBUG_MODE=1 to see the full error.

21:20:06.496 | ERROR   | prefect_aws.observers.ecs - ECS observer task crashed

websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 503

Yeah, that's probably a one-off. What version of

prefect

is your worker running? We've shipped some additional stability enhancements for WebSockets in recent releases.

•

prefect_aws-0.7.5

Hmm, I'll have to check if that

websockets.exceptions.InvalidStatus

error is slipping past our error handling.

@alex it's happening again - websocket issue and sqs queue grows

@Lee Trout What version of

prefect

are you running?

same as always - pulling latest of the prefect worker image whenever the container starts

Not sure if useful but this is my Marvin chat from last week regarding this: https://prefect-community.slack.com/archives/C079VLLH5D3/p1775734470899749