Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

*Graceful shutdown of jobs in k8s*
We’re using Prefect to run flows of varying lengths on a k8s cluster with autoscaling.
Our current approach is to try and let flows complete before terminating a node (setting a very long grace period for the job).
However, it seems like `capture_sigterm` in prefect.engine aborts the task without any option for overriding it, so the flow ends up dying the moment the node is marked for termination.
Did anyone successfully implement something similar? any ideas?

note:  overriding the signal from within the flow isn’t always possible since sometimes it’s not ran on the main thread