Yes — this looks like a Prefect library bug, not something specific to your code. What’s happening The worker’s reconnect logic uses exponential backoff. In the buggy code path, it computes something like:

base_delay * 2 ** (attempt - 1)

If the worker keeps failing to reconnect for a long time,

attempt

can grow very large. Eventually that exponent gets big enough that Python raises:

OverflowError: int too large to convert to float

So the immediate cause is usually: - the worker repeatedly losing its connection to Prefect for a long stretch - the reconnect attempt counter continuing to increase - the backoff calculation overflowing before it gets capped So your instinct is right: this is in library code. Good news: it has already been fixed upstream A fix was merged in Prefect that caps the exponent before it can overflow: PR #22086 The fix is intended for Prefect 3.7.3+. If your image includes Prefect

<= 3.7.2

, you can still hit this. What you can do now 1. Check the actual Prefect version inside the container

python -c "import prefect; print(prefect.__version__)"

2. Upgrade to a version that includes the fix once available in the image you use. Since you’re on

prefecthq/prefect-aws:0.7.9-python3.12

, the collection tag does not necessarily make the Prefect core version obvious, so checking in-container is the safest route. 3. Investigate why reconnects are failing repeatedly, because that’s what triggers the bug in the first place. Common causes: - intermittent network path from ECS task to Prefect API / Cloud - DNS or TLS issues - temporary Prefect API unavailability - auth/token issues causing repeated reconnect failures 4. Short-term mitigation - let ECS restart the worker if it crashes - monitor for repeated connection failures before the overflow - if possible, reduce long-lived broken connectivity conditions rather than leaving the worker in a reconnect loop indefinitely If you need a place to raise it Since this is already fixed upstream, I’d first recommend upgrading when the fixed release is available. If you still see it after upgrading to a fixed Prefect version, then open a GitHub issue here: Prefect issues You could mention the existing fix/PR for context: PR #22086 Bottom line - Yes, this is a Prefect bug. - It is triggered by many consecutive reconnect failures. - It has already been fixed upstream. - Best path: check your Prefect version and upgrade to 3.7.3+ when available. If you want, I can also help you figure out which Prefect version is bundled in

prefecthq/prefect-aws:0.7.9-python3.12

and suggest the cleanest upgrade path for ECS.

If there are no preceding worker log messages, then it may just mean the reconnect loop isn’t surfacing the underlying connection exception clearly before the overflow kills the process. A few practical ways to check: 1. Turn up Prefect logging verbosity on the worker Set a more verbose log level for the ECS worker task, for example:

PREFECT_LOGGING_LEVEL=DEBUG

That may expose the reconnect failures before the crash. If you’re setting env vars in the ECS task definition, add that there and redeploy the worker. 2. Confirm the worker can reach the Prefect API from inside the task From the running ECS container, test:

python -c "import os; print(os.environ.get('PREFECT_API_URL'))"

Then try reaching it directly, for example with

curl

if available:

curl -I "$PREFECT_API_URL"

Or with Python if

curl

isn’t installed:

python - <<'PY'
import os, urllib.request
url = os.environ["PREFECT_API_URL"]
print("Testing:", url)
try:
    with urllib.request.urlopen(url, timeout=10) as r:
        print("status:", r.status)
except Exception as e:
    print("error:", repr(e))
PY

If you’re using Prefect Cloud, you should at least be able to reach the API endpoint over HTTPS. 3. Verify credentials/environment are stable Check whether these are always present and valid in the worker container: -

PREFECT_API_URL

-

PREFECT_API_KEY

for Cloud, if applicable You can inspect whether they exist without printing secrets:

python - <<'PY'
import os
for key in ["PREFECT_API_URL", "PREFECT_API_KEY"]:
    print(key, "set" if os.getenv(key) else "missing")
PY

4. Check ECS/container-level signals Since the worker is running on ECS, the problem may be outside Prefect itself: - task restarts due to memory pressure / OOM - transient network issues in the VPC/subnets - DNS resolution failures - egress restrictions via security groups / NAT / proxy - periodic credential refresh issues if using IAM/task roles Useful places to inspect: - ECS task stopped reason - CloudWatch logs for the container - VPC Flow Logs if enabled - DNS/network health for the subnets/NAT path 5. Watch for long API gaps from the Prefect side If this is a worker losing contact with the API, you may see symptoms such as: - worker goes offline unexpectedly in the UI - delayed polling for work - gaps in heartbeats / late run submission If you’re using Prefect Cloud, check whether the worker appears to disappear and reappear around the crash times. 6. Add your own connectivity probe alongside the worker If the logs are too sparse, a simple sidecar or periodic script can test connectivity to the API every 30–60s and log failures. For example:

python - <<'PY'
import os, time, urllib.request, traceback

url = os.environ["PREFECT_API_URL"]

while True:
    try:
        with urllib.request.urlopen(url, timeout=10) as r:
            print(f"OK status={r.status}")
    except Exception as e:
        print("CONNECTIVITY ERROR:", repr(e))
        traceback.print_exc()
    time.sleep(30)
PY

That won’t fix the issue, but it can help prove whether there are intermittent network/API failures around the same time. 7. Upgrade anyway Even if you don’t find the underlying disconnect, I’d still recommend upgrading once the fixed release is available, because the worker should not crash from a backoff overflow in the first place. Fix reference: PR #22086 My honest take Given that: - the error happens only occasionally - there are no other obvious app-level errors - the traceback is entirely in reconnect logic this most likely points to an intermittent transport/API disconnect that is not being logged well enough before the overflow occurs. If you want, I can help you with a concrete ECS debugging checklist for this worker, including exactly which env vars to add and what to inspect in CloudWatch/ECS.