I might be missing the context—can you share what you’re trying to solve? A few details will help me get you an exact answer: - What are you doing with Prefect (e.g., creating a deployment, running a flow, configuring a worker)? - Prefect version (2.x or 3.x) and Cloud vs. open-source - How you’re running it (local, Docker, Kubernetes) and work pool/worker type if applicable - The exact error/traceback and any commands you ran - A minimal code snippet (flow + deployment config if relevant) - Output of

prefect version

If it’s easier, paste the error and the few lines around where it occurs—I can usually spot the issue quickly.

Short answer: already-running flows are fine; you won’t “lose” orchestration data. The impact is mostly on runs that were about to start and on your pool’s concurrency. What happens when an ECS worker OOMs/restarts in Prefect 3.6.16 (Cloud, Hybrid): - Runs already submitted to infrastructure - Unaffected. Once a flow run is launched (e.g., as its own ECS task), it reports state/logs directly to Prefect Cloud. The worker is just a submitter/poller. - Scheduled runs the worker hadn’t touched yet - Unaffected. They remain SCHEDULED and the next healthy worker can pick them up. - Runs moved to PENDING but not yet submitted - These can be left behind in PENDING if the worker dies abruptly (OOM means it can’t set CRASHED). PENDING runs are not auto-reset to SCHEDULED by the server. They’ll sit PENDING until you reschedule/cancel them. - PENDING counts toward your work pool and work queue concurrency. Enough stuck PENDING runs can “soak up” your pool limit and make the system look stalled. Data loss considerations: - Flow logs/states: No loss for runs that were already executing; they send logs/states directly to Cloud. - Worker logs: You may lose the worker’s own logs around the time of the OOM. - Pending runs: Not data loss, but they won’t progress until rescheduled, and they hold concurrency capacity while PENDING. Operational tips for your scenario (80k runs, pool limit 400, worker at 3GB): - Cap per-worker concurrency - Start workers with a limit so they don’t try to submit too many runs at once:

prefect worker start --pool <your-pool> --limit 50

- Scale out with multiple workers instead of one very “wide” worker. - Find and clear orphaned PENDING runs (if any) - In the UI, filter Flow Runs by state=PENDING and take action (reschedule or cancel + re-run). - Or use a quick script to reset specific runs from PENDING back to SCHEDULED:

import asyncio, datetime
    from prefect import states
    from prefect.client.orchestration import get_client

    async def reschedule(flow_run_id: str):
        async with get_client() as client:
            await client.set_flow_run_state(
                flow_run_id,
                state=states.Scheduled(
                    scheduled_time=datetime.datetime.utcnow(),
                    message="Rescheduled after worker OOM"
                ),
            )

    asyncio.run(reschedule("<FLOW_RUN_ID>"))

- Right-size the worker - 3GB may be tight under heavy load. Either increase memory for the worker service or reduce per-worker concurrency (

--limit

) and add additional workers to spread load. - Poll settings (optional tuning) - Defaults are generally fine; if needed you can adjust worker poll interval/prefetch: -

PREFECT_WORKER_QUERY_SECONDS

(default ~10s) -

PREFECT_WORKER_PREFETCH_SECONDS

(default ~10s) - See Settings & Profiles. References: - Workers overview: https://docs.prefect.io/v3/concepts/workers/ - Worker CLI (includes

--limit

): https://docs.prefect.io/v3/api-ref/cli/worker - Work pools and concurrency: https://docs.prefect.io/v3/concepts/work-pools/#concurrency-limits If you’d like, share a screenshot/count of PENDING runs in that pool and your current worker

--limit

(if any). I can help you pick a safe per-worker limit and give you a small script to bulk reschedule any stuck PENDING runs.

Great question. In short: it’s not that ECS “needs” more memory—it's that the Prefect ECS worker is likely handling much higher concurrency per process than your EKS worker, which drives Python memory up. A few specifics to explain the difference: Why ECS worker memory spikes - Concurrency, not CPU: The worker is an async submitter. If you don’t cap it, it will try to submit as many runs as are available (bounded by pool/queue limits). With a pool limit of 400 and a massive backlog (80k runs), one ECS worker can easily try to juggle hundreds of submissions concurrently and hold lots of run metadata, request payloads, and boto3/client objects at once. - PENDING counts: Work pool concurrency counts PENDING + RUNNING. If one worker moves a large batch to PENDING concurrently, you get a big in-memory wave while it renders infra and calls ECS RunTask for each. - ECS SDK overhead: The ECS worker uses boto3/botocore. Large numbers of concurrent AWS calls (task definitions/overrides, env, IAM/config) create more Python objects than the Kubernetes client typically does at the same scale. With 400+ concurrent submissions, this can be the difference between 512MB and multiple GB. Why EKS “works fine” at 512MB - Likely lower per-worker concurrency: Many Kubernetes setups run multiple small workers or naturally limit concurrency (either by

--limit

, queue limits, or cluster policies). If your K8s worker submits fewer runs at a time, it uses far less memory. What to do (works well on ECS) - Set a per-worker concurrency cap. This is the single most effective control.

prefect worker start --pool <your-pool> --limit 25

- Then scale out workers horizontally (e.g., 8 workers x 25 = 200 total), instead of one worker trying to drive the entire 400. This makes memory predictable. - Keep an eye on PENDING runs. If a worker OOMs mid-submission, runs may stick in PENDING and tie up your pool limit. Clear or reschedule any orphans so they don’t “soak” capacity. - Optional tuning: - Poll interval/prefetch can be adjusted to smooth spikes: -

PREFECT_WORKER_QUERY_SECONDS

(default ~10s) -

PREFECT_WORKER_PREFETCH_SECONDS

(default ~10s) - Prefer several smaller workers (each 512MB–1GB) to a single very large one. How to confirm this is your case - Check the worker logs during OOM periods: how many flow runs was it submitting at once? - In the UI, look at concurrent PENDING + RUNNING in that pool when the worker is healthy. - If you’re starting the worker without

--limit

, add it and watch memory flatten. Docs for reference - Workers overview (includes

--limit

): https://docs.prefect.io/v3/concepts/workers/ - Work pools and concurrency: https://docs.prefect.io/v3/concepts/work-pools/#concurrency-limits - Worker CLI: https://docs.prefect.io/v3/api-ref/cli/worker If you can share: - Your current ECS worker start command (including flags), - Launch type (Fargate vs EC2), - Approximate concurrent runs shown in the pool when it OOMs, …I can suggest a concrete

--limit

and worker count that will keep memory under 512MB–1GB per worker.