Not sure if this is the right channel to post this but prefect cloud seems to be very unresponsive/slow in the last few hours. We aren't able to deploy any new flows and having problems scheduling existing ones. Wanted to know if anyone at @Prefect has any insights regarding the problem. We have some critical jobs production jobs scheduled on prefect.

I'm also seing some timeout when trying to register flows?

File "register_flows.py", line 11, in _register_flow

flow_id = prefect_client.register(

File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 938, in register

self.graphql(

File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 443, in graphql

result = <http://self.post|self.post>(

File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 398, in post

response = self._request(

File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 633, in _request

response = self._send_request(

File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 497, in _send_request

response = <http://session.post|session.post>(

File "/root/miniconda3/lib/python3.8/site-packages/requests/sessions.py", line 577, in post

return self.request('POST', url, data=data, json=json, **kwargs)

File "/root/miniconda3/lib/python3.8/site-packages/requests/sessions.py", line 529, in request

resp = self.send(prep, **send_kwargs)

File "/root/miniconda3/lib/python3.8/site-packages/requests/sessions.py", line 645, in send

r = adapter.send(request, **kwargs)

File "/root/miniconda3/lib/python3.8/site-packages/requests/adapters.py", line 519, in send

raise ConnectionError(e, request=request)

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)"))

Yeah, we are seeing similar timeouts. And scheduled flows aren't running as per schedule and they show up in the web UI after missing a few runs.

Thanks for the report! We’re investigating

@jawnsy also seeing jobs 30+ minutes late while agent is actively querying

We are also experiencing this on prefect cloud, since about 9 AM UTC if that helps

Are you still seeing issues? We saw high CPU pressure in our GraphQL services, which we are continuing to investigate, but in the meantime, the load seems to have come down and our latency & request rate metrics are returning to normal

my most-recent job (12:00 CST) ran on time

my flow runs are still stuck in pending

Im still getting timeouts trying to deploy new flows

I've also seen timeout errors when triggering flows from other flows (prefect cloud v1). any updates on this?

We’re not aware of any ongoing issues with the service. Can you DM me with some details of your tenant ID or flow so that we can investigate?