I am getting this error only when I run Prefect-Server in the cloud, I am running the server in GCP as a container in a Container Optimized OS

asyncio.exceptions.TimeoutError
04:20:16.970 | WARNING | prefect.server.services.marklateruns - MarkLateRuns took 5.009215 seconds to run, which is longer than its loop interval of 5.0 seconds.
04:20:17.195 | ERROR   | prefect.server.services.failexpiredpauses - Unexpected error in: TimeoutError()
Traceback (most recent call last):

asyncio.exceptions.TimeoutError
07:48:18.910 | WARNING | prefect.server.services.flowrunnotifications - FlowRunNotifications took 5.011953 seconds to run, which is longer than its loop interval of 4 seconds.
07:48:18.917 | ERROR   | prefect.server.services.recentdeploymentsscheduler - Unexpected error in: TimeoutError()

hmm I don't see a default setting value of 4 seconds either could you try setting your

PREFECT_API_SERVICES_SCHEDULER_LOOP_SECONDS

to a higher number of seconds? the recent deployment scheduler appears to be falling back to the setting here

should I try manually setting it to 60?

actually it looks like the timeout is coming from exceeding this value

PREFECT_API_SERVICES_PAUSE_EXPIRATIONS_LOOP_SECONDS

which appears to have a default of 5 you haven't set this value explicitly have you? I can ask internally for recommendations here

no, I have not

hmm i see this now - but can't say I'm sure why everything is timing out like that when on your COS VM 🤔

What is the recommend way to run prefect opensource in the cloud?

i would say that in general we recommend the helm chart, but I understand that's not an option for everyone. to be honest working with COS VMs is not something I have a lot of experience with, but it seems like one way or another the loop services are having a hard time talking to the DB. are you using postgres? if you opt for a traditional VM it might make sense to use docker-compose to manage things

Yes, we are using postgres

yes! you could use something like CloudSQL

@Nate I wanted to update you, I was able to finally resolve the issue. It ended up being a firewall rule that was preventing the server from correctly connecting to the db

glad you got it resolved, thanks for sharing your solution!