Olivér Atanaszov
01/03/2023, 5:47 PMFlow(..., on_failure=cleanup_fn)
does not trigger the function for cancellation. Oh and I'm using Prefect 1.0.Bianca Hoch
01/03/2023, 11:21 PMBianca Hoch
01/03/2023, 11:22 PMOlivér Atanaszov
01/04/2023, 9:49 AMOlivér Atanaszov
01/04/2023, 9:56 AMKubernetesRun
on different nodes:
def cleanup(flow, state):
# cleanup resources in K8s
# kubectl delete jobs --all --namespace $USER
with Flow("myflow", ..., on_failure=cleanup) a flow:
# do some work, e.g. run various child flow runs on K8s
create_flow_run("kubernetes_job")
create_flow_run("kubernetes_job")
# ...
Now, if I cancel the "main" flow named "myflow" in this example, I would like to run the cleanup
function to make sure all resources are released.
Does this make sense?Bianca Hoch
01/09/2023, 9:56 PM@resource_manager
decorator? It is a speciality task which allows for custom setup and cleanup of cloud resources like kubernetes deployments (more docs on that here, if needed). There is an example of how to use that in the article posted above as well.Bianca Hoch
01/09/2023, 10:24 PMon_failure
flow parameter included in the code snippet you provided is called when a flow enters a failed state, and not a Cancelled
state. You could try creating your resource_manager
task, passing it to a state handler function which checks to see if the the flow is_finished
(finished captures the Cancelled
state), and then passing the state handler to the flow.Bianca Hoch
01/09/2023, 10:34 PMOlivér Atanaszov
01/10/2023, 3:29 PM