hi prefect, we had the following error last night

prefect\client\base.py", line 130, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Server error '502 Bad Gateway' for url
'<https://api.prefect.cloud/api/accounts/xxxxx/flow_runs/filter>'
For more information check: <https://httpstatuses.com/502> 
Agent stopped!

which we believe is caused by a failure to communicate with the cloud instance. We have a default argument for number of tries, which is set here to 3 https://github.com/PrefectHQ/prefect/blob/42e5a978390575b8bd6ef4c258ba703f7e28fca2/src/prefect/utilities/services.py#L18 - ideally we would like to be able to reconfigure it or set it to infinite so that when our agents cannot communicate with cloud, they don't all collapse - we had to restart everything this morning. As another point, this obviously generated hundreds of late flows. Why is there no 'select all' function on the UI, or capability to remove all late flows. The best I have been able to do so far is to go through and manually click to remove all late flows. Feel like there should be more functionality for actions on batches of flow runs

I had this same thing happen to one of our flow runs last night. As to mass flow cancel, retry etc, one can do it via API. Some example code (cancels all late flows) is here