https://prefect.io logo
z

Zachary Lee

07/12/2023, 8:13 PM
I'm seeing a lot of warnings about CancellationCleanup loop service taking a long time in my prefect server logs:
Copy code
20:03:50.981 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 30.438319 seconds to run, which is longer than its loop interval of 20.0 seconds.
20:04:24.279 | WARNING | prefect.server.services.cancellationcleanup - CancellationCleanup took 33.296899 seconds to run, which is longer than its loop interval of 20.0 seconds.
I've already deleted all the
Cancelled
runs in the hopes of fixing this, but it seems to have no effect. Also, my server's CPU usage is very high, and I'm wondering if it's potentially related to this. Does anyone know what makes CancellationCleanup take so long?
I do confirm that my cpu usage goes way down (800m on kubernetes to <100m) when I turn off the CancellationCleanup service via: https://docs.prefect.io/latest/api-ref/prefect/settings/#prefect.settings.PREFECT_API_SERVICES_CANCELLATION_CLEANUP_ENABLED What would make this service be that CPU intensive and take that long? I've deleted all my Cancelled runs
6 Views