@Marvin How can we prevent that prefect flows or the pods running the flows are killed when running a helm upgrade?

It seems there isn't a direct reference on preventing Prefect flows or the pods running them from being killed during a

helm upgrade

. However, here's a general approach to handle upgrades on Kubernetes without disrupting running pods: 1. Use a

PodDisruptionBudget

(PDB): This ensures that a certain number or percentage of pods remain available during maintenance events, such as upgrades.

yaml
   apiVersion: policy/v1
   kind: PodDisruptionBudget
   metadata:
     name: my-prefect-pdb
   spec:
     minAvailable: 1
     selector:
       matchLabels:
         app: prefect

2. Set proper

terminationGracePeriodSeconds

: This allows pods extra time to shut down properly. You can define this in your Helm chart values or directly in your manifest files.

yaml
   spec:
     terminationGracePeriodSeconds: 60

3. Rolling Updates: Ensure that your Helm chart and Kubernetes deployment strategy use rolling updates to prevent downtime.

yaml
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxUnavailable: 1

4. Job Restart Policy: If you are using Kubernetes jobs for your flow runs, set

restartPolicy

to

OnFailure

.

yaml
   spec:
     restartPolicy: OnFailure

5. Monitor with Liveness and Readiness Probes: Use health checks to manage the state of your pods more effectively. For Prefect-specific configurations tailored to your deployments, I recommend referring to the Prefect Kubernetes deployment guide for more details. Feel free to implement the above general Kubernetes practices or dive deeper into specific configurations from the Prefect documentation and adjust your setup accordingly.