Nikhil Jain
02/03/2023, 7:21 PMPrefect 2.0
where runs get started but don’t finish properly and the UI says Submission timed out.
Error on UI:
Submission failed. RuntimeError: Timed out after 120.6031973361969s while watching task for status {until_status or 'STOPPED'}
AWS Cloudwatch Logs from the flow run: indicate that the task runner stops immediately after starting the run and does not execute any of the tasks in the Flow. (sharing logs in the next message in thread)./usr/local/lib/python3.9/runpy.py:127: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
warn(RuntimeWarning(msg))
17:02:41.063 | DEBUG | prefect.client - Connecting to API at <https://api.prefect.cloud/api/accounts/adee12e2-25d9-428a-9a77-32d8d7ca2ac7/workspaces/28562586-0812-4e92-803f-99ce64384cbd/>
17:02:41.448 | DEBUG | Flow run 'stirring-cheetah' - Loading flow for deployment 'staging-ecs'...
17:02:41.825 | DEBUG | Flow run 'stirring-cheetah' - Starting 'ConcurrentTaskRunner'; submitted tasks will be run concurrently...
17:02:41.826 | DEBUG | prefect.task_runner.concurrent - Starting task runner...
17:02:41.834 | DEBUG | prefect.client - Connecting to API at <https://api.prefect.cloud/api/accounts/adee12e2-25d9-428a-9a77-32d8d7ca2ac7/workspaces/28562586-0812-4e92-803f-99ce64384cbd/>
17:02:42.453 | DEBUG | prefect.task_runner.concurrent - Shutting down task runner...
17:02:42.455 | INFO | prefect.engine - Engine execution of flow run '1c1c05a9-bf85-483b-a115-ba77f7a6350e' aborted by orchestrator: This run has already terminated.
17:02:42.464 | DEBUG | prefect.client - Connecting to API at <https://api.prefect.cloud/api/accounts/adee12e2-25d9-428a-9a77-32d8d7ca2ac7/workspaces/28562586-0812-4e92-803f-99ce64384cbd/>
Luca Vehbiu
04/12/2023, 5:09 PM