https://prefect.io logo
Title
b

Bernard Greyling

05/05/2022, 6:17 AM
Morning All We are using prefect via a kubernetes agent. We've noticed that failed flows don't clean up prefect jobs and in some cases the dask-excecutor. Is there any passive mechanism to clean up these dead jobs ?
etl                prefect-job-29a6d24a-hwnzx                         0/1     ImagePullBackOff             0          6d20h
etl                prefect-job-30bd9ad7-zsc6z                         0/1     CreateContainerConfigError   0          11h
etl                prefect-job-39468921-lt95t                         0/1     CreateContainerConfigError   0          6d20h
etl                prefect-job-3c1aee81-hvbv4                         0/1     Completed                    0          7h49m
etl                prefect-job-3c84d152-wfqw9                         0/1     Completed                    0          7h53m
etl                prefect-job-3fb2397e-7dc5b                         0/1     ImagePullBackOff             0          11h
etl                prefect-job-5ba165eb-rfxhq                         0/1     CreateContainerConfigError   0          6d20h
etl                prefect-job-62ae93a3-ptbcr                         0/1     CreateContainerConfigError   0          6d20h
etl                prefect-job-6cfd964c-p7xpl                         0/1     ImagePullBackOff             0          6d20h
etl                prefect-job-86e07407-nmpwb                         0/1     ImagePullBackOff             0          6d20h
etl                prefect-job-9478c771-bfg26                         0/1     ImagePullBackOff             0          11h
etl                prefect-job-950ba45b-tv4f2                         0/1     CreateContainerConfigError   0          11h
etl                prefect-job-a199b1d6-6vwmd                         0/1     Completed                    0          7h49m
etl                prefect-job-a4976ce3-gxqv6                         0/1     Completed                    0          6h5m
a

Anna Geller

05/05/2022, 10:13 AM
the easiest way to configure that would be by setting this environment variable in your Kubernetes agent deployment spec:
export DELETE_FINISHED_JOBS=True
but it is by default True
b

Bernard Greyling

05/05/2022, 10:15 AM
Excellent! Thanks Anna, I'll have a look
👍 1
m

Mathijs Carlu

06/01/2022, 8:47 AM
Hi @Anna Geller, does prefect orion have something similar?
a

Anna Geller

06/01/2022, 9:39 AM
not yet, but let me open an issue for tracking
:upvote: 1
@Marvin open "Add a flag to
KubernetesFlowRunner
to allow cleaning up finished Kubernetes jobs"