@Marvin, does prefect server v3.0 support graceful shutdowns? We are running prefect server on EKS and sometimes observe

Job reached backoff limit

. It seems to be related to karpenter reshuffling pods and nodes ...

Prefect Server v3.0 does support graceful shutdowns, particularly in response to SIGTERM signals. The

task_worker.py

component of Prefect includes a mechanism to handle SIGTERM signals, initiating a graceful shutdown of the task worker. This is typically achieved by registering a signal handler that logs the shutdown initiation and calls the appropriate stop functions. However, there have been issues reported with Prefect workers not ending flows gracefully, especially when transitioning from Prefect Agents to Workers. This might be relevant in your case, where pod reshuffling by Karpenter on EKS is causing jobs to reach their backoff limits. It might be beneficial to investigate this further to ensure that the Prefect workers are correctly handling shutdowns and that any necessary configurations for graceful termination are in place. For more details, you can explore the following resources: - Prefect's

task_worker.py

source code - GitHub issue on Prefect worker not ending flow gracefully: Issue #15033 If you're using EKS with Karpenter, ensure that your pods are configured with appropriate termination grace periods to allow Prefect to handle shutdowns gracefully.