Morning All
We are using prefect via a kubernetes ...
# ask-communityb
Bernard Greyling
05/05/2022, 6:17 AMMorning All
We are using prefect via a kubernetes agent.
We've noticed that failed flows don't clean up prefect jobs and in some cases the dask-excecutor.
Is there any passive mechanism to clean up these dead jobs ?
Copy code
etl prefect-job-29a6d24a-hwnzx 0/1 ImagePullBackOff 0 6d20h
etl prefect-job-30bd9ad7-zsc6z 0/1 CreateContainerConfigError 0 11h
etl prefect-job-39468921-lt95t 0/1 CreateContainerConfigError 0 6d20h
etl prefect-job-3c1aee81-hvbv4 0/1 Completed 0 7h49m
etl prefect-job-3c84d152-wfqw9 0/1 Completed 0 7h53m
etl prefect-job-3fb2397e-7dc5b 0/1 ImagePullBackOff 0 11h
etl prefect-job-5ba165eb-rfxhq 0/1 CreateContainerConfigError 0 6d20h
etl prefect-job-62ae93a3-ptbcr 0/1 CreateContainerConfigError 0 6d20h
etl prefect-job-6cfd964c-p7xpl 0/1 ImagePullBackOff 0 6d20h
etl prefect-job-86e07407-nmpwb 0/1 ImagePullBackOff 0 6d20h
etl prefect-job-9478c771-bfg26 0/1 ImagePullBackOff 0 11h
etl prefect-job-950ba45b-tv4f2 0/1 CreateContainerConfigError 0 11h
etl prefect-job-a199b1d6-6vwmd 0/1 Completed 0 7h49m
etl prefect-job-a4976ce3-gxqv6 0/1 Completed 0 6h5ma
Anna Geller
05/05/2022, 10:13 AM
Yes, there is a flag on the Kubernetes agent called `delete_finished_jobs`: https://github.com/PrefectHQ/prefect/blob/40e2e2bd1c1de5651bc7fb484c823e43cfc69877/src/prefect/agent/kubernetes/agent.py#L84
Anna Geller
05/05/2022, 10:14 AM
the easiest way to configure that would be by setting this environment variable in your Kubernetes agent deployment spec:
Copy code
export DELETE_FINISHED_JOBS=Trueb
Bernard Greyling
05/05/2022, 10:15 AMExcellent! Thanks Anna, I'll have a look
👍 1
m
Mathijs Carlu
06/01/2022, 8:47 AMHi @Anna Geller, does prefect orion have something similar?
a
Anna Geller
06/01/2022, 9:39 AM
not yet, but let me open an issue for tracking
upvote 1
Anna Geller
06/01/2022, 9:41 AM
@Marvin open "Add a flag to
KubernetesFlowRunnerm
Marvin
06/01/2022, 9:41 AM11 Views