I also see this error in my server logs:

│   File "/usr/local/lib/python3.12/site-packages/prefect/server/events/ordering.py", line 201, in preceding_event_confirmed                                                              │
│     await handler(waiter, depth + 1)                                                                                                                                                    │
│   File "/usr/local/lib/python3.12/site-packages/prefect/server/events/triggers.py", line 483, in reactive_evaluation                                                                    │
│     async with AsyncExitStack() as stack:                                                                                                                                               │
│                ^^^^^^^^^^^^^^^^                                                                                                                                                         │
│   File "/usr/local/lib/python3.12/contextlib.py", line 754, in __aexit__                                                                                                                │
│     raise exc_details[1]                                                                                                                                                                │
│   File "/usr/local/lib/python3.12/contextlib.py", line 737, in __aexit__                                                                                                                │
│     cb_suppress = await cb(*exc_details)                                                                                                                                                │
│                   ^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                │
│   File "/usr/local/lib/python3.12/contextlib.py", line 217, in __aexit__                                                                                                                │
│     await anext(self.gen)                                                                                                                                                               │
│   File "/usr/local/lib/python3.12/site-packages/prefect/server/events/ordering.py", line 201, in preceding_event_confirmed                                                              │
│     await handler(waiter, depth + 1)                                                                                                                                                    │
│   File "/usr/local/lib/python3.12/site-packages/prefect/server/events/triggers.py", line 485, in reactive_evaluation                                                                    │
│     await stack.enter_async_context(                                                                                                                                                    │
│   File "/usr/local/lib/python3.12/contextlib.py", line 659, in enter_async_context                                                                                                      │
│     result = await _enter(cm)                                                                                                                                                           │
│              ^^^^^^^^^^^^^^^^                                                                                                                                                           │
│   File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__                                                                                                               │
│     return await anext(self.gen)                                                                                                                                                        │
│            ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                        │
│   File "/usr/local/lib/python3.12/site-packages/prefect/server/events/ordering.py", line 176, in preceding_event_confirmed                                                              │
│     raise MaxDepthExceeded(event)                                                                                                                                                       │
│ prefect.server.events.ordering.MaxDepthExceeded: occurred=DateTime(2024, 12, 4, 17, 52, 25, 134884, tzinfo=Timezone('UTC')) event='prefect.flow-run.Cancelling' resource=Resource(root= │
│ {'prefect.state-name': 'Cancelling', 'prefect.state-type': 'CANCELLING',

which was released in 3.1.3 https://github.com/PrefectHQ/prefect/releases?page=2

I think i had some slight issues with the database upgrade, but after it failed once weirdly running it again the second time worked. Is there anything i might be doing that could cause that this? Should i try wiping the postgres db and redeploying?

I tried resetting the database with

prefect server database reset

but that actually failed with trying to downgrade, with alembic missing a revision

In the end i just manually dropped all the tables in my prefect-dev environment, and restarted the server, giving that a bit of time to see if its worked, but so far looks promising for my dev prefect server

Sadly that didn't work 😢 this is becoming more painful than our monorepo pydantic v2 upgrade. Current issues i'm working through: • Prefect server issues - what seems to stem from slow server - database comms:

File "/usr/local/lib/python3.12/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 949, in connect
    await_only(creator_fn(*arg, **kw)),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only
    return current.parent.switch(awaitable)  # type: ignore[no-any-return,attr-defined] # noqa: E501
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn
    value = await result
            ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/asyncpg/connection.py", line 2420, in connect
    async with compat.timeout(timeout):
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/timeouts.py", line 115, in __aexit__
    raise TimeoutError from exc_val
TimeoutError
13:54:15.592 | WARNING | prefect.server.services.flowrunnotifications - FlowRunNotifications took 6.797844 seconds to run, which is longer than its loop interval of 4 seconds.
13:54:16.315 | WARNING | prefect.server.services.recentdeploymentsscheduler - RecentDeploymentsScheduler took 7.518834 seconds to run, which is longer than its loop interval of 5 seconds.

I've checked and the load on the db is nominal, number of DB connections is <10% the max allowed, trying to think of more things to investigate @Nate if you have any tips or guidance i'd massively appreciate it! We're running these versions for client and server (although the helm chart i used is technically the latest - 3.1.5)

Version:             3.1.4
API version:         0.8.4
Python version:      3.12.7
Git commit:          78ee41cb
Built:               Wed, Nov 20, 2024 7:37 PM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         server
Pydantic version:    2.10.3
Integrations:
  prefect-slack:     0.3.1
  prefect-github:    0.3.1
  prefect-aws:       0.5.2