https://prefect.io logo
Title
k

karteek

08/30/2022, 3:53 PM
Not sure if this is the right channel to post this but prefect cloud seems to be very unresponsive/slow in the last few hours. We aren't able to deploy any new flows and having problems scheduling existing ones. Wanted to know if anyone at @Prefect has any insights regarding the problem. We have some critical jobs production jobs scheduled on prefect.
t

Thomas Frederik Hoeck

08/30/2022, 3:55 PM
I'm also seing some timeout when trying to register flows?
File "register_flows.py", line 11, in _register_flow
flow_id = prefect_client.register(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 938, in register
self.graphql(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 443, in graphql
result = <http://self.post|self.post>(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 398, in post
response = self._request(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 633, in _request
response = self._send_request(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 497, in _send_request
response = <http://session.post|session.post>(
File "/root/miniconda3/lib/python3.8/site-packages/requests/sessions.py", line 577, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)"))
k

karteek

08/30/2022, 3:58 PM
Yeah, we are seeing similar timeouts. And scheduled flows aren't running as per schedule and they show up in the web UI after missing a few runs.
j

jawnsy

08/30/2022, 4:02 PM
Thanks for the report! We’re investigating
d

David Michael Carter

08/30/2022, 4:31 PM
@jawnsy also seeing jobs 30+ minutes late while agent is actively querying
k

kevin

08/30/2022, 4:50 PM
We are also experiencing this on prefect cloud, since about 9 AM UTC if that helps
j

jawnsy

08/30/2022, 5:47 PM
Are you still seeing issues? We saw high CPU pressure in our GraphQL services, which we are continuing to investigate, but in the meantime, the load seems to have come down and our latency & request rate metrics are returning to normal
d

David Michael Carter

08/30/2022, 5:49 PM
my most-recent job (12:00 CST) ran on time
k

kevin

08/30/2022, 6:00 PM
my flow runs are still stuck in pending
sorry, to be more concise, they're in a scheduled but late flow run state and not showing up when the agent queries but the agent is green
k

Kingsley

08/30/2022, 6:03 PM
Im still getting timeouts trying to deploy new flows
b

Bob De Schutter

09/05/2022, 10:48 AM
I've also seen timeout errors when triggering flows from other flows (prefect cloud v1). any updates on this?
j

jawnsy

09/05/2022, 3:27 PM
We’re not aware of any ongoing issues with the service. Can you DM me with some details of your tenant ID or flow so that we can investigate?