Oof—that message usually means the underlying container died before the flow could complete. The phrase “Essential container in task exited (Error Code: 1)” is AWS ECS’s wording, so I’m guessing you’re using an ECS work pool/worker. Can you confirm your infrastructure (ECS Fargate vs EC2, region, work pool name)? Also, did you kick off many runs at once and/or change your image tag recently (e.g., using “latest”)?
Quickest way to pinpoint the cause
1) Open one failed flow run in Prefect Cloud and copy the ECS Task ARN (Infra section) if present.
2) In AWS console: ECS → your cluster → Tasks → find the stopped task for that run.
- Check:
- Stopped reason (e.g., OutOfMemoryError, CannotPullContainerError)
- Container exit code (will show 1 here)
- Events tab for image pull/network errors
- Logs tab (links to CloudWatch) for the actual traceback or OOM kill
3) Also check your Prefect worker logs around the time the runs started.
Common causes when runs suddenly start failing after increasing concurrency
- OOM (most common): The container is getting killed due to memory pressure when more run concurrently.
- Fix: Increase task memory/CPU in your ECS work pool job configuration OR reduce concurrency.
- Image pull problems:
- Docker Hub rate limits when spinning up many tasks at once (CannotPullContainerError: toomanyrequests)
- Wrong/mutated image tag or ECR permission issues
- Missing/incorrect env vars or credentials:
- e.g., PREFECT_API_URL/PREFECT_API_KEY not present at runtime, or secrets not injected
- Networking:
- Fargate subnets without NAT → container can’t reach Prefect Cloud
- Security groups blocking egress
- IP exhaustion in crowded subnets when many tasks start at once
- Entry point/command failures:
- If your image or entrypoint changed, the startup command may be exiting immediately
What to check right now
- Get the ECS task’s Stopped reason and a few lines from CloudWatch logs and share them here—those are the fastest clues.
- Inspect the current job configuration for your work pool to confirm CPU/memory, image, env vars:
prefect work-pool inspect "<your-pool-name>"
- If you burst many runs at once, consider temporarily capping concurrency:
prefect work-pool set-concurrency-limit "<your-pool-name>" 2
Then rerun a single failed flow to see if it succeeds with less contention.
Likely remediations based on what you find
- If OOM: raise memory (and possibly CPU) in the ECS job configuration for that pool; or reduce parallelism.
- If CannotPullContainerError: verify the image exists and credentials/permissions are correct; avoid Docker Hub “latest” under load, prefer ECR with a pinned tag.
- If auth errors in logs: ensure PREFECT_API_URL and PREFECT_API_KEY are set in the work pool default job variables and/or deployment overrides.
- If networking errors: ensure private subnets have NAT, SG allows egress, and the task role allows what it needs.
Docs that may help
- ECS guide (setup, job configuration, logging, troubleshooting):
Prefect ECS guide
- States and crashes:
Prefect states
- Add logging to your flows:
Flow logging
- Work pool CLI reference:
Work pool CLI
- Concurrency limits overview:
Concurrency limits
If you can share:
- Work pool type (ECS Fargate/EC2), region
- Approx. number of runs started at once
- Image repo:tag
- ECS task Stopped reason and the first ~50 lines from CloudWatch logs
…I can help you zero in on the exact fix.