Thanks for all the detail — this helps a ton. What’s going on - The SQS queue you’re seeing is for the ECS observer, not for work dispatch. In Prefect 3.x, workers pull scheduled flow runs from the Prefect API (polling). SQS is only used by the ECS observer to receive ECS task events (RUNNING/STOPPED/crash detection, log forwarding). - The websocket warning is the key: when the worker/observer cannot open the wss connection to Prefect Cloud’s events endpoint, it cannot publish events. In that state, the observer will keep accumulating events locally and will also avoid deleting the corresponding SQS messages (so they can be replayed once connectivity returns). - That explains both symptoms: the SQS queue grows and the container’s memory footprint grows with it. When you redeploy, the connection is healthy for a while, the observer drains SQS and flushes events, then the connection fails again and the cycle repeats. - Jobs still start because scheduling is independent of the SQS observer and websocket. Most likely cause - A network/proxy/firewall issue on egress to wss://api.prefect.cloud over 443 (or a proxy that blocks Authorization headers on websocket upgrade). The “timed out during opening handshake” message is consistent with this. Immediate mitigations (to stop OOM while you fix network) - Cap the in-process events queue so it can’t grow unbounded:

PREFECT_EVENTS_WORKER_MAX_QUEUE_SIZE=50000

Note: if the connection stays down, older event data may be dropped once the cap is hit, but your worker will remain healthy. - If you can accept losing ECS crash detection/log forwarding temporarily, disable the ECS observer to stop producing/queuing those events:

PREFECT_INTEGRATIONS_AWS_ECS_OBSERVER_ENABLED=false

Work dispatch will continue; you’ll just lose ECS crash observability until re-enabled. - Bake prefect-aws into your image instead of pip installing on boot to reduce container churn:

# In your Dockerfile
RUN pip install prefect-aws
# Then just run:
prefect worker start --pool our-workpool --type ecs

How to fix the root cause (wss connectivity) - Turn on debug to see the underlying socket/TLS/proxy error:

PREFECT_DEBUG_MODE=1

This will include the exception that is causing the handshake timeout. - Verify egress: - Allow outbound 443 to api.prefect.cloud. - If you use a corporate proxy, ensure websocket upgrades are allowed and that Authorization headers are forwarded for websocket handshakes. If your proxy strips Authorization on upgrade, the connection will fail. - If you must use a proxy, set appropriate env vars (HTTPS_PROXY/NO_PROXY). Consider adding api.prefect.cloud to NO_PROXY if the proxy blocks websockets. - Sanity checks from the worker container: - Confirm your API base is correct (used to derive the wss URL):

prefect config view | grep PREFECT_API_URL

- Quick TLS reachability check (not a websocket test, but useful):

curl -I <https://api.prefect.cloud/>

- If possible in your environment, temporarily test a simple websocket client against the events URL using your API key to confirm upgrades are permitted. If that’s not feasible, the PREFECT_DEBUG_MODE trace usually reveals the culprit (proxy reset, cert verification, etc.). AWS/SQS specifics to review - In CloudWatch for the SQS queue, check AgeOfOldestMessage: it should rise when you see the websocket warnings and then fall after a restart (when connectivity returns). - Ensure the queue’s visibility timeout is comfortably larger than the observer’s processing window and network roundtrips (long polling is used, but this usually isn’t the limiting factor here). Version info request - Which versions are you on for prefect and prefect-aws? There have been improvements to the ECS observer and event clients in recent 3.x releases. Sharing: - output of

prefect version

- output of

python -c "import prefect_aws, pkgutil;print(prefect_aws.__version__)"

Summary of recommended next steps 1) Enable PREFECT_DEBUG_MODE=1 and capture the full websocket error from the worker logs. 2) Verify/allow outbound websocket (wss) to api.prefect.cloud with Authorization headers intact (proxy/firewall). 3) Set PREFECT_EVENTS_WORKER_MAX_QUEUE_SIZE to cap memory while you work the network issue. 4) Optionally disable the ECS observer temporarily (PREFECT_INTEGRATIONS_AWS_ECS_OBSERVER_ENABLED=false) if you need immediate relief and can tolerate reduced crash observability. 5) Bake prefect-aws into the image to avoid pip installs at runtime. If you can share the debug log snippet and your versions, I can help you pinpoint the exact failure mode and suggest the minimal network/proxy change to resolve it.

Short answer: yes, that can happen. Why - In Prefect 3.x, the ECS observer deletes SQS messages right after it pulls them from the queue. That deletion does not depend on the websocket to Prefect Cloud being up or on successful event emission. So even if the wss connection is down, many messages will still be drained. - If you’re seeing a subset that “sticks,” it’s usually due to one of these: - The observer received the message but hit an error before the delete call (network hiccup on DeleteMessage, container restart, or event loop blocked). Those messages reappear after the visibility timeout and accumulate. - Rarely, malformed SQS messages without a ReceiptHandle are skipped and not deleted. - IAM issues for sqs:DeleteMessage (intermittent or missing permission on the queue ARN). - The observer is pointed at the right queue, but processing gets backlogged (e.g., heavy memory pressure, retry loops elsewhere), so batches that arrive during those gaps sit until the next healthy window. This can look like “only some types” are stuck. What to check next - Turn on debug to surface SQS errors alongside the websocket warnings:

PREFECT_DEBUG_MODE=1

Look for any “delete_message”/SQS-related exceptions in the worker logs. - Verify SQS metrics in CloudWatch: - NumberOfMessagesReceived vs NumberOfMessagesDeleted (they should track closely) - AgeOfOldestMessage (will climb during backlogs) - Confirm the task role attached to the worker has, at minimum, these permissions on the queue ARN: - sqs:GetQueueUrl, sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility - Make sure you’re pointing at the intended queue:

PREFECT_INTEGRATIONS_AWS_ECS_OBSERVER_SQS_QUEUE_NAME=<your-queue>
PREFECT_INTEGRATIONS_AWS_ECS_OBSERVER_SQS_QUEUE_REGION=<region>

- If you need to keep the service stable while debugging the websocket/network issue: - Cap the in-process event buffer so memory can’t grow unbounded:

PREFECT_EVENTS_WORKER_MAX_QUEUE_SIZE=50000

- If acceptable, temporarily disable the ECS observer to stop consuming/producing those events:

PREFECT_INTEGRATIONS_AWS_ECS_OBSERVER_ENABLED=false

- Optionally scale the worker service to 2+ tasks to increase SQS consumption while you investigate. Notes on versions - Your test across latest and 3.4.20 matches the implementation — the SQS deletion behavior has been consistent across those versions. The websocket warning you’re seeing is a known symptom of network/proxy issues (an example of the warning appearing in the wild: issue #16299), but the root cause in that ticket is different from yours. If you can share: - one “stuck” message body (redacted) from SQS, - a snippet of logs with PREFECT_DEBUG_MODE=1 around the time the queue starts growing, - and confirm the IAM policy on the worker task role, …I can help pinpoint whether you’re hitting delete failures, processing gaps, or unrelated messages landing in the queue.