Hi everyone, we use Prefect Cloud and have many runs scheduled for many flows on many projects, and everything has been working fine for a few months now, which is great! But this morning one, and only one, scheduled run failed, and upon inspection of the Agent Logs I found that it failed because it got a “read timeout”, as per the log message below:

WARNING:urllib3.connectionpool:Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)")': /

it got those many times, then the job was considered dead and etc. Does anyone know why this could have happened?

Can you tell what time exactly did it happen? You can tell the UTC time

around 8:56 AM this morning

I checked it and the API had some latency during that time so it looks like it’s not your fault at all, it’s the API’s fault. But everything should be resolved now.

ahh that’s great to know! Thank you very much!

I can ask but not sure we can share such information for security/privacy reasons

are outages/latency issues like this reported anywhere? we missed several SLAs this morning for this reason

We have prefect.status.io

looks like last nights issue wasn't reported