Hello, does anyone currently experience issues with the flow run in Prefect Cloud? All our flow runs are stuck in submitted state but not running, across all agents and environments.

Hi @Hui Zheng , do you by any chance have more than 625 late runs? Our work queue pauses over that number and so no new runs will get picked up.

we also seeing currently-running run failed due to some connection issue

[4 May 2021 6:00pm]: HTTPSConnectionPool(host='10.124.0.1', port=443): Max retries exceeded with url: /apis/batch/v1/namespaces/default/jobs (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fe1cc52e630>: Failed to establish a new connection: [Errno 110] Connection timed out',))

We’re looking into this

Thank you. We traced the errors to GCP k8e cluster. It seems there are some infrastructure issue with GCP.

Hey @Hui Zheng, everything looks healthy on our end. This looks like a kubernetes issue.