<@ULVA73B9P> I am randomly getting “State message:...
# ask-marvin
e
@Marvin I am randomly getting “State message: Flow run could not be submitted to infrastructure” on my AWS ECS task for my nightly flows, any suggestions?
m
thought for 110.2 seconds
That message usually means Prefect successfully tried to hand the flow run to ECS, but ECS could not start the task. In other words, this is typically an infrastructure submission/startup issue, not a failure in your flow code. A few common causes for random/intermittent ECS failures: Most likely - Transient ECS/Fargate capacity issues - Image pull failures (
CannotPullContainerError
) - Docker Hub/ECR rate limiting - Networking issues with subnets / ENIs / security groups - IAM permission issues for the execution role - CloudWatch logging permission issues if log configuration is enabled A few concrete things to check: 1. Look at the worker logs The ECS worker usually logs the real AWS exception that gets summarized into
Flow run could not be submitted to infrastructure
. 2. Look at the ECS task stop reason In the AWS ECS console, inspect the failed task and check: -
stoppedReason
-
stopCode
- container
reason
If you see things like: -
TaskFailedToStart
-
CannotPullContainerError
-
ResourceInitializationError
-
RESOURCE:MEMORY
/
RESOURCE:CPU
those usually identify the root cause. 3. Check whether the image pull is flaky This is very common for nightly bursts. - If you are pulling from Docker Hub, rate limits can cause intermittent failures. - If using private ECR/GHCR, verify auth and permissions. Relevant examples: - Docker Hub rate limit causing this exact Prefect/ECS message - CannotPullContainerError example - Private registry credentials discussion 4. Check CloudWatch log permissions If CloudWatch logging is enabled for the ECS task, missing
logs:*
permissions on the task execution role can prevent task startup. Example: - CloudWatch permission issue causing submission failure 5. Check capacity / networking around the nightly run time - Are many tasks starting at once? - Are subnets running out of IPs / ENIs? - Are you using
FARGATE_SPOT
and seeing occasional placement failures? Useful docs: - Prefect ECS worker docs - Manual ECS deployment / troubleshooting - Work pools and infrastructure concepts A practical short checklist: - Check the exact ECS stopped reason for one failed run - Check whether all failures happen at the same time window nightly - Confirm what image registry you use (Docker Hub, ECR, GHCR, etc.) - Confirm whether CloudWatch logging is enabled - See if the worker logs show retries or AWS API errors If you want, send me: - the full error text from one failed run - the ECS task stoppedReason - whether you’re using Fargate or EC2 launch type - which image registry you use and I can help narrow it down quickly.