Yes — based on the code, the retry overflow bug is real, and your description matches it very closely. What I could verify: Issue 1:

/workers/connect

getting

1013 service_unavailable

- I could confirm the worker channel treats this as a retryable connection failure. -

1013

is effectively “try again later”, so the worker keeps reconnecting rather than treating it as fatal. - In newer Prefect 3.x worker-channel code, there is also a REST fallback path when the websocket channel is unhealthy, so the worker can continue polling without the bidirectional websocket. Issue 2: exponential backoff overflow killing the worker - I could verify this is a real bug in the worker channel reconnect logic. - The problem is exactly what you described: the exponential term was allowed to grow before the cap was applied, so after enough retries the calculation overflows before

min(...)

can save it. - The buggy pattern was effectively:

min(base * 2**attempt, max_delay)

instead of capping the exponent / short-circuiting before the multiplication. Fix status - I found a fix merged in Prefect upstream: PR

#22086

- The fix caps the exponent before multiplication / returns the max delay early. - The research result indicates: -

3.7.2

is still vulnerable - the fix is on

main

/

3.7.3.dev1+

- it should be resolved in the next stable release after that I wasn’t able to independently verify GitHub issue/discussion links because GitHub search was unavailable from my tools at the moment, so I can’t give you a canonical issue URL for those reports right now. Practical workarounds 1. Upgrade to a version containing the fix - Best workaround if possible. - If you can run a dev build or the next patch release after

3.7.2

, that should address the overflow crash. 2. Let the worker use REST fallback - If you’re on a recent 3.x version with worker-channel fallback support, the worker should continue via REST polling when websocket connectivity is unhealthy. - Watch worker logs for messages indicating the websocket channel is unhealthy and fallback is active. 3. Add process-level resilience in ECS - Since your current behavior is “overflow → crash → ECS restart → repeat”, keep the ECS restart policy in place as a temporary mitigation. - Also make sure logs are shipped to CloudWatch so you can see whether the websocket never recovers or whether fallback is engaging. 4. Reduce duration of uninterrupted failure if you patch locally - If you vendor/patch Prefect internally, the safe logic is to cap the exponent before computing

2**n

. Conceptually:

exponent = min(attempt - 1, SAFE_MAX)
delay = min(base * 2**exponent, max_delay)

or even better:

if exponent >= SAFE_MAX:
    return max_delay

What this means for your ECS deployment - If you’re seeing constant

1013 service_unavailable

, that is likely a service-side or connectivity-pressure condition rather than a misbehavior in your worker itself. - The *worker should survive it*; the fact that it eventually dies is the separate retry-overflow bug. - So there are really two problems: 1. websocket channel unavailable / degraded 2. reconnect loop overflow bug turning a recoverable outage into a worker crash Recommended next step - If you tell me the exact Prefect version your worker image is using, I can help you determine whether you’re on a vulnerable build and suggest the safest upgrade path. - I can also help you identify whether your version should already support REST fallback and what log lines to look for.