My ECSTasks are timing out after they reach PENDING status e

Question

My ECSTasks are timing out after they reach PENDING status even after I tried adding ECS full access My setup guide was the Prefect 2 Fargate guide on Medium using Prefect 2 7 12 What might be the cause of this

James Gatter · Answer

```15 18 59 006 | INFO | prefect agent Submitting flow run f2d5d977 88e2 4a90 af08 b07e224728f1 15 18 59 309 | INFO | prefect infrastructure process Opening process tested butterfly 15 18 59 368 | INFO | prefect agent Completed submission of flow run f2d5d977 88e2 4a90 af08 b07e224728f1 usr local lib python3 9 runpy py 127 RuntimeWarning prefect engine found in sys modules after import of package prefect but prior to execution of prefect engine this may result in unpredictable behaviour warn RuntimeWarning msg 15 19 01 792 | INFO | Flow run tested butterfly Downloading flow code from storage at 15 19 02 755 | INFO | Flow run tested butterfly Network ip 10 0 30 30 ec2 internal Instance Linux 5 10 162 141 675 amzn2 x86 64 x86 64 with glibc2 31 Agent is healthy 15 19 02 757 | INFO | Flow run tested butterfly Python = 3 9 16 API 0 8 4 Prefect = 2 7 12 15 19 02 869 | INFO | Flow run tested butterfly Finished in state Completed 15 19 03 624 | INFO | prefect infrastructure process Process tested butterfly exited cleanly 15 19 46 465 | INFO | prefect agent Submitting flow run 25532c7f c80b 423c 99b3 d716de85712f 15 19 46 823 | INFO | prefect infrastructure ecs task ECSTask statuesque grasshopper Retrieving task definition prefect quilt s3 cyflows 15 19 46 864 | INFO | prefect infrastructure ecs task ECSTask statuesque grasshopper Registering task definition 15 19 47 279 | INFO | prefect infrastructure ecs task ECSTask statuesque grasshopper Creating task run 15 19 47 903 | INFO | prefect infrastructure ecs task ECSTask statuesque grasshopper Waiting for task run to start 15 19 47 924 | INFO | prefect infrastructure ecs task ECSTask statuesque grasshopper Status is PROVISIONING 15 19 57 970 | INFO | prefect infrastructure ecs task ECSTask statuesque grasshopper Status is PENDING 15 21 48 479 | ERROR | prefect agent Failed to submit flow run 25532c7f c80b 423c 99b3 d716de85712f to infrastructure Traceback most recent call last File usr local lib python3 9 site packages prefect agent py line 476 in submit run and capture errors result = await infrastructure run task status=task status File usr local lib python3 9 site packages prefect aws ecs py line 610 in run = await run sync in worker thread File usr local lib python3 9 site packages prefect utilities asyncutils py line 91 in run sync in worker thread return await anyio to thread run sync File usr local lib python3 9 site packages anyio to thread py line 31 in run sync return await get asynclib run sync in worker thread File usr local lib python3 9 site packages anyio backends asyncio py line 937 in run sync in worker thread return await future File usr local lib python3 9 site packages anyio backends asyncio py line 867 in run result = context run func args File usr local lib python3 9 site packages prefect aws ecs py line 805 in create task and wait for start self wait for task start File usr local lib python3 9 site packages prefect aws ecs py line 1048 in wait for task start for task in self watch task run File usr local lib python3 9 site packages prefect aws ecs py line 1033 in watch task run raise RuntimeError RuntimeError Timed out after 120 57538270950317s while watching task for status until status or STOPPED 15 21 48 481 | INFO | prefect agent Completed submission of flow run 25532c7f c80b 423c 99b3 d716de85712f ```

James Gatter · Answer

I ve tried three different flow deployments pushed using the GitHub action in that workflow and no dice

James Gatter · Answer

All my flows use a custom image built on the Prefect official that is about 4 GB It takes about 4 minutes to pull Could this be the explanation for the timeout

James Gatter · Answer

My agent on ECS has `Task CPU 4 vCPU Task memory 8 GB` so it probably can t be a resource bound containment but possibly a time bound one

Ryan Peden · Answer

The 4 minute pull time sounds like the cause The ECSTask block waits for 2 minutes by default but the timeout is adjustable If you open the block in the UI you ll see a Task Watch Timeout Seconds field you can edit

Ryan Peden · Answer

Screenshot 2023 02 17 at 3 36 56 PM png

James Gatter · Answer

Surprised I never saw that option thanks Giving it a go now

James Gatter · Answer

It worked Thanks so much

Ryan Peden · Answer

You re welcome I m happy to hear it worked