https://prefect.io logo
Title
c

Cody

04/27/2023, 12:05 AM
Hello, I'm trying to go through the ECS guide here - https://prefecthq.github.io/prefect-aws/#getting-started , but my Flow Runs always get stuck in
Pending
before timing out after 120 seconds. I can see the Flow Run start and hit
Pending
and at the same time I see the ECS Task appear in the AWS console (shown in screenshot) - My Agent logs the following:
17:59:41.311 | INFO    | prefect.agent - Submitting flow run 'efc6b670-c269-4a87-ba2f-467e4a2c6e85'
17:59:42.349 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Retrieving task definition 'prefect__ecs-task-flow__ecs-task-deployment'...
17:59:43.619 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Creating task run...
17:59:44.216 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Waiting for task run to start...
17:59:44.299 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Status is PROVISIONING.
17:59:54.508 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'berserk-copperhead': Status is PENDING.
18:01:46.542 | ERROR   | prefect.agent - Failed to submit flow run 'efc6b670-c269-4a87-ba2f-467e4a2c6e85' to infrastructure.
Traceback (most recent call last):
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect\agent.py", line 490, in _submit_run_and_capture_errors
    result = await infrastructure.run(task_status=task_status)
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 619, in run
    ) = await run_sync_in_worker_thread(
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect\utilities\asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 814, in _create_task_and_wait_for_start
    self._wait_for_task_start(
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 1057, in _wait_for_task_start
    for task in self._watch_task_run(
  File "C:\Users\Cody\anaconda3\envs\prefect2\lib\site-packages\prefect_aws\ecs.py", line 1042, in _watch_task_run
    raise RuntimeError(
RuntimeError: Timed out after 122.3249180316925s while watching task for status {until_status or 'STOPPED'}
18:01:46.561 | INFO    | prefect.agent - Completed submission of flow run 'efc6b670-c269-4a87-ba2f-467e4a2c6e85'
Does anyone know what the issue could be? I'm fairly new to ECS, so I'm trying to figure out if the fix I need is in the ECS or Prefect side of things. My ECS Cluster is brand new, I haven't made any changes - basically its just what you'd get if you hit next at every screen during the creation process.