Bruno Murino

    Bruno Murino

    8 months ago
    Hi everyone, we use Prefect Cloud and have many runs scheduled for many flows on many projects, and everything has been working fine for a few months now, which is great! But this morning one, and only one, scheduled run failed, and upon inspection of the Agent Logs I found that it failed because it got a “read timeout”, as per the log message below:
    WARNING:urllib3.connectionpool:Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)")': /
    it got those many times, then the job was considered dead and etc. Does anyone know why this could have happened?
    Anna Geller

    Anna Geller

    8 months ago
    Can you tell what time exactly did it happen? You can tell the UTC time
    Bruno Murino

    Bruno Murino

    8 months ago
    around 8:56 AM this morning
    Anna Geller

    Anna Geller

    8 months ago
    I checked it and the API had some latency during that time so it looks like it’s not your fault at all, it’s the API’s fault. But everything should be resolved now.
    Bruno Murino

    Bruno Murino

    8 months ago
    ahh that’s great to know! Thank you very much!
    do we know what caused the latency?
    Anna Geller

    Anna Geller

    8 months ago
    I can ask but not sure we can share such information for security/privacy reasons
    Zach Schumacher

    Zach Schumacher

    8 months ago
    are outages/latency issues like this reported anywhere? we missed several SLAs this morning for this reason
    for example, confluent and databricks have a slack app you can install that publishes those kinds of messages to our alerts channels internally
    Anna Geller

    Anna Geller

    8 months ago
    Zach Schumacher

    Zach Schumacher

    8 months ago
    looks like last nights issue wasn't reported
    but the status page is good to know about, thanks for sharing!