Nikhil Jain

02/03/2023, 7:21 PM
I am seeing some sporadic errors on
Prefect 2.0
where runs get started but don’t finish properly and the UI says Submission timed out. Error on UI:
Submission failed. RuntimeError: Timed out after 120.6031973361969s while watching task for status {until_status or 'STOPPED'}
AWS Cloudwatch Logs from the flow run: indicate that the task runner stops immediately after starting the run and does not execute any of the tasks in the Flow. (sharing logs in the next message in thread).
Flow run logs:
/usr/local/lib/python3.9/ RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
17:02:41.063 | DEBUG   | prefect.client - Connecting to API at <>
17:02:41.448 | DEBUG   | Flow run 'stirring-cheetah' - Loading flow for deployment 'staging-ecs'...
17:02:41.825 | DEBUG   | Flow run 'stirring-cheetah' - Starting 'ConcurrentTaskRunner'; submitted tasks will be run concurrently...
17:02:41.826 | DEBUG   | prefect.task_runner.concurrent - Starting task runner...
17:02:41.834 | DEBUG   | prefect.client - Connecting to API at <>
17:02:42.453 | DEBUG   | prefect.task_runner.concurrent - Shutting down task runner...
17:02:42.455 | INFO    | prefect.engine - Engine execution of flow run '1c1c05a9-bf85-483b-a115-ba77f7a6350e' aborted by orchestrator: This run has already terminated.
17:02:42.464 | DEBUG   | prefect.client - Connecting to API at <>

Luca Vehbiu

04/12/2023, 5:09 PM
also experiencing this