Cody
04/27/2023, 12:05 AMPending
before timing out after 120 seconds.
I can see the Flow Run start and hit Pending
and at the same time I see the ECS Task appear in the AWS console (shown in screenshot) -
My Agent logs the following:
17:59:41.311 | INFO | prefect.agent - Submitting flow run 'efc6b670-c269-4a87-ba2f-467e4a2c6e85'
17:59:42.349 | INFO | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Retrieving task definition 'prefect__ecs-task-flow__ecs-task-deployment'...
17:59:43.619 | INFO | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Creating task run...
17:59:44.216 | INFO | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Waiting for task run to start...
17:59:44.299 | INFO | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Status is PROVISIONING.
17:59:54.508 | INFO | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Status is PENDING.
18:01:46.542 | ERROR | prefect.agent - Failed to submit flow run 'efc6b670-c269-4a87-ba2f-467e4a2c6e85' to infrastructure.
Traceback (most recent call last):
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect\agent.py", line 490, in _submit_run_and_capture_errors
result = await infrastructure.run(task_status=task_status)
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 619, in run
) = await run_sync_in_worker_thread(
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect\utilities\asyncutils.py", line 91, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
result = context.run(func, *args)
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 814, in _create_task_and_wait_for_start
self._wait_for_task_start(
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 1057, in _wait_for_task_start
for task in self._watch_task_run(
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 1042, in _watch_task_run
raise RuntimeError(
RuntimeError: Timed out after 122.3249180316925s while watching task for status {until_status or 'STOPPED'}
18:01:46.561 | INFO | prefect.agent - Completed submission of flow run 'efc6b670-c269-4a87-ba2f-467e4a2c6e85'
Does anyone know what the issue could be? I'm fairly new to ECS, so I'm trying to figure out if the fix I need is in the ECS or Prefect side of things.
My ECS Cluster is brand new, I haven't made any changes - basically its just what you'd get if you hit next at every screen during the creation process.