Cody
04/27/2023, 12:05 AMPending
before timing out after 120 seconds.
I can see the Flow Run start and hit Pending
and at the same time I see the ECS Task appear in the AWS console (shown in screenshot) -
My Agent logs the following:
17:59:41.311 | INFO | prefect.agent - Submitting flow run 'efc6b670-c269-4a87-ba2f-467e4a2c6e85'
17:59:42.349 | INFO | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Retrieving task definition 'prefect__ecs-task-flow__ecs-task-deployment'...
17:59:43.619 | INFO | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Creating task run...
17:59:44.216 | INFO | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Waiting for task run to start...
17:59:44.299 | INFO | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Status is PROVISIONING.
17:59:54.508 | INFO | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Status is PENDING.
18:01:46.542 | ERROR | prefect.agent - Failed to submit flow run 'efc6b670-c269-4a87-ba2f-467e4a2c6e85' to infrastructure.
Traceback (most recent call last):
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect\agent.py", line 490, in _submit_run_and_capture_errors
result = await infrastructure.run(task_status=task_status)
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 619, in run
) = await run_sync_in_worker_thread(
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect\utilities\asyncutils.py", line 91, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
result = context.run(func, *args)
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 814, in _create_task_and_wait_for_start
self._wait_for_task_start(
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 1057, in _wait_for_task_start
for task in self._watch_task_run(
File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 1042, in _watch_task_run
raise RuntimeError(
RuntimeError: Timed out after 122.3249180316925s while watching task for status {until_status or 'STOPPED'}
18:01:46.561 | INFO | prefect.agent - Completed submission of flow run 'efc6b670-c269-4a87-ba2f-467e4a2c6e85'
Does anyone know what the issue could be? I'm fairly new to ECS, so I'm trying to figure out if the fix I need is in the ECS or Prefect side of things.
My ECS Cluster is brand new, I haven't made any changes - basically its just what you'd get if you hit next at every screen during the creation process.Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by