https://prefect.io logo
Title
j

Joshua Greenhalgh

11/18/2022, 12:02 PM
Is there an issue with the cloud API today? I have agent running which has suddenly (since ~9.20 UTC) stopped picking up flow runs - the agent is up and running - it did have the following error (but killed the agent and restarted since)
WARNING:urllib3.connectionpool:Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)")': /
Any help would be much apreciated...
Was running successfully just before this with following log;
[2022-11-18 09:25:32,514] INFO - agent | Completed deployment of flow run 2a48c870-57c3-4950-9ea4-11c4cf0ea0a9
INFO:agent:Completed deployment of flow run 2a48c870-57c3-4950-9ea4-11c4cf0ea0a9
Cloud UI seems to still be querying the agent successfully;
Ok they literally just all started again...have never seen behaviour like this before...
Things seem to be working again but everything takes a good while to be scheduled - seeing warnings like this;
2022-11-18 12:18:24+0000] WARNING - prefect.wait_for_flow_run | Flow '6af7dc87-0af4-45b9-bf49-283d7a54206d-cloudwatch_logs_ingestion_flow': It has been 15 seconds and your flow run has not been submitted by an agent. Agent a6356184-c1c3-4dc6-b48b-5b061c791f57 (agent) has matching labels and last queried a few seconds ago. It should deploy your flow run.
k

Kalise Richmond

11/18/2022, 5:00 PM
Hi @Joshua Greenhalgh, happy to hear things are working again. You can always check our API status at https://prefect.status.io/ I'm not entirely sure when that warning pops up but the first thing that comes to mind is if you have concurrency limits on your agent label and it is waiting to submit the flow?