Triet Le
05/09/2024, 9:02 AM08:48:43.955 | WARNING | prefect.server.services.recentdeploymentsscheduler - RecentDeploymentsScheduler took 11.444709 seconds to run, which is longer than its loop interval of 5 seconds.
08:48:44.291 | WARNING | prefect.server.services.flowrunnotifications - FlowRunNotifications took 18.066717 seconds to run, which is longer than its loop interval of 4 seconds.
08:48:44.681 | WARNING | prefect.server.services.failexpiredpauses - FailExpiredPauses took 12.970546 seconds to run, which is longer than its loop interval of 5.0 seconds.
08:48:44.767 | WARNING | prefect.server.services.marklateruns - MarkLateRuns took 15.615843 seconds to run, which is longer than its loop interval of 5.0 seconds.
08:48:49.936 | WARNING | prefect.server.services.failexpiredpauses - FailExpiredPauses took 5.247492 seconds to run, which is longer than its loop interval of 5.0 seconds.
08:48:50.155 | WARNING | prefect.server.services.flowrunnotifications - FlowRunNotifications took 5.831411 seconds to run, which is longer than its loop interval of 4 seconds.
08:48:50.472 | ERROR | prefect.server.services.marklateruns - Unexpected error in: TimeoutError()
On some occasions it’s okay to leave the error and move on. Howerver, my Prefect server keeps crashing every 2-3days and I have to manually SSH to the server and restart. Despite attempts to adjust job schedules and minimize simultaneous job executions, this error still persist.
Can anyone helps giving me a pointer on this? Thanks!