Hi team, just dropping here an issue I'm experiencing on our Prefect Server deployed on premise, but I'm not able to provide a minimal reproducible example for 😞, but maybe just looking at the exception may help you direct me to a relevant issue already open. This is something we are experiencing after upgrading Prefect to 2.13.5. Basically, I have a flow executing the same task many times with different inputs. After a while, I start getting the following error on the agent side:

Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<http://prefect-server:4200/api/task_runs/90e9a53e-56b6-40b9-b8e7-290c37439597/set_state>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>

And the following on the server side:

01:00:57.358 | ERROR   | prefect.server - Error running before-transition hook in rule <class 'prefect.server.orchestration.core_policy.SecureTaskConcurrencySlots'>: !TimeoutError()

Here is our Prefect version dump:

Version:             2.13.5
API version:         0.8.4
Python version:      3.9.18
Git commit:          0fe4277b
Built:               Thu, Oct 5, 2023 3:58 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         server

I'm also attaching the full stack trace I'm seeing on the server side. Any clue what may be going on? Should I open an issue for it?

Hey Giorgio! Are you using tag-based concurrency on your tasks? This looks to me like a resource contention issue within your DB - perhaps check out any metrics on the number of locks that are being held / CPU consumption?

Hi Chris, yes I'm using it indeed. Yes I agree it could be something like that, I was just making sure it wasn't some known issue that was solved by further prefect updates. I'll have a look and come back here in case I found anything more. Thanks!

You could try refactoring your concurrency that is set on task tags to a more explicit approach using our global concurrency utilities (documented here: https://docs.prefect.io/latest/guides/global-concurrency-limits/). My guess is that you will see some improvements

Thank you Chris, I saw a new tab in the UI about it, but didn’t have the opportunity to dig deep, will do and let you know!