https://prefect.io logo
Title
n

Nace Plesko

01/10/2023, 12:36 AM
Hi! I'm using prefect v1 and I'm trying to activate zombie killer to clean up tasks that are failed but are stuck in forever running state. From what I understood from this link, all I need to do to activate it is to add
env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"}
to my run config. I added that but the tasks are still being stuck. Is there anything else I need to configure in order to activate zombie killer?
c

Christopher Boyd

01/10/2023, 2:39 PM
Where do the tasks run?
n

Nace Plesko

01/10/2023, 2:39 PM
As kubernetes jobs
On GCP
c

Christopher Boyd

01/10/2023, 2:40 PM
This is a common issue - if a task fails or gets restarted for any reason in kubernetes, Kubernetes natively willr estart the pod, but Prefect is not aware of this restart so it continues to see a task as “running”
https://github.com/PrefectHQ/prefect/issues/7116 is for Prefect 2, but applicable as well for Prefect 1
in short, there is not a good answer for this behavior at this moment
n

Nace Plesko

01/10/2023, 2:42 PM
I see, thanks!
m

Michael Levenson

01/10/2023, 4:26 PM
Hi also having issues with this - using spot IO on top. How can we get this fixed asap. Affects production pipelines that we have. Huge Blocker