@Marvin i'm running prefect in kubernetes (GKE). after a few days i notice that tasks take a lot longer to run even though there doesn't seem to be any cpu/memory/io contention. It may have nothing to do with prefect itself, but i wondered if this is possibly somehow related to prefect itself

It seems there's no direct evidence of Prefect causing performance degradation over time in Kubernetes environments like GKE. However, here are a few aspects you might consider: 1. Concurrency Limits: Prefect allows setting concurrency limits with a slot decay mechanism. If configured, this could inadvertently slow down task execution over time. You might want to check if this is affecting your tasks. More on concurrency limits. 2. Infrastructure Setup: Ensure your Kubernetes setup is optimized for running Prefect tasks. Prefect's documentation provides guidance on running flows on Kubernetes, which might help in ensuring that your infrastructure isn't inadvertently causing slowdowns. Kubernetes setup guide. 3. General 3.x Gotchas: Ensure that your setup aligns with Prefect 3.x changes, such as the transition from agents to workers and other infrastructure changes. This may not directly cause performance issues but ensuring compatibility can prevent unexpected behavior. If none of these seem to be the issue, it might be worth investigating other aspects of your Kubernetes environment, such as resource limits or scaling configurations. If you have any specific configurations or use cases, feel free to share, and I can look into those further.