Morning All We are using prefect via a kubernetes ...
# prefect-community
b
Morning All We are using prefect via a kubernetes agent. We've noticed that failed flows don't clean up prefect jobs and in some cases the dask-excecutor. Is there any passive mechanism to clean up these dead jobs ?
Copy code
etl                prefect-job-29a6d24a-hwnzx                         0/1     ImagePullBackOff             0          6d20h
etl                prefect-job-30bd9ad7-zsc6z                         0/1     CreateContainerConfigError   0          11h
etl                prefect-job-39468921-lt95t                         0/1     CreateContainerConfigError   0          6d20h
etl                prefect-job-3c1aee81-hvbv4                         0/1     Completed                    0          7h49m
etl                prefect-job-3c84d152-wfqw9                         0/1     Completed                    0          7h53m
etl                prefect-job-3fb2397e-7dc5b                         0/1     ImagePullBackOff             0          11h
etl                prefect-job-5ba165eb-rfxhq                         0/1     CreateContainerConfigError   0          6d20h
etl                prefect-job-62ae93a3-ptbcr                         0/1     CreateContainerConfigError   0          6d20h
etl                prefect-job-6cfd964c-p7xpl                         0/1     ImagePullBackOff             0          6d20h
etl                prefect-job-86e07407-nmpwb                         0/1     ImagePullBackOff             0          6d20h
etl                prefect-job-9478c771-bfg26                         0/1     ImagePullBackOff             0          11h
etl                prefect-job-950ba45b-tv4f2                         0/1     CreateContainerConfigError   0          11h
etl                prefect-job-a199b1d6-6vwmd                         0/1     Completed                    0          7h49m
etl                prefect-job-a4976ce3-gxqv6                         0/1     Completed                    0          6h5m
a
the easiest way to configure that would be by setting this environment variable in your Kubernetes agent deployment spec:
Copy code
export DELETE_FINISHED_JOBS=True
but it is by default True
b
Excellent! Thanks Anna, I'll have a look
👍 1
m
Hi @Anna Geller, does prefect orion have something similar?
a
not yet, but let me open an issue for tracking
upvote 1
@Marvin open "Add a flag to
KubernetesFlowRunner
to allow cleaning up finished Kubernetes jobs"