Eric Sexton
07/06/2026, 7:38 PMMarvin
07/06/2026, 7:38 PMMarvin
07/06/2026, 7:39 PMCannotPullContainerError)
- Docker Hub/ECR rate limiting
- Networking issues with subnets / ENIs / security groups
- IAM permission issues for the execution role
- CloudWatch logging permission issues if log configuration is enabled
A few concrete things to check:
1. Look at the worker logs
The ECS worker usually logs the real AWS exception that gets summarized into Flow run could not be submitted to infrastructure.
2. Look at the ECS task stop reason
In the AWS ECS console, inspect the failed task and check:
- stoppedReason
- stopCode
- container reason
If you see things like:
- TaskFailedToStart
- CannotPullContainerError
- ResourceInitializationError
- RESOURCE:MEMORY / RESOURCE:CPU
those usually identify the root cause.
3. Check whether the image pull is flaky
This is very common for nightly bursts.
- If you are pulling from Docker Hub, rate limits can cause intermittent failures.
- If using private ECR/GHCR, verify auth and permissions.
Relevant examples:
- Docker Hub rate limit causing this exact Prefect/ECS message
- CannotPullContainerError example
- Private registry credentials discussion
4. Check CloudWatch log permissions
If CloudWatch logging is enabled for the ECS task, missing logs:* permissions on the task execution role can prevent task startup.
Example:
- CloudWatch permission issue causing submission failure
5. Check capacity / networking around the nightly run time
- Are many tasks starting at once?
- Are subnets running out of IPs / ENIs?
- Are you using FARGATE_SPOT and seeing occasional placement failures?
Useful docs:
- Prefect ECS worker docs
- Manual ECS deployment / troubleshooting
- Work pools and infrastructure concepts
A practical short checklist:
- Check the exact ECS stopped reason for one failed run
- Check whether all failures happen at the same time window nightly
- Confirm what image registry you use (Docker Hub, ECR, GHCR, etc.)
- Confirm whether CloudWatch logging is enabled
- See if the worker logs show retries or AWS API errors
If you want, send me:
- the full error text from one failed run
- the ECS task stoppedReason
- whether you’re using Fargate or EC2 launch type
- which image registry you use
and I can help narrow it down quickly.Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by