https://prefect.io logo
Title
o

Olivér Atanaszov

01/03/2023, 5:47 PM
Hey! I would like to run a cleanup function whenever the flow run is cancelled from the Cloud UI, but apparently using
Flow(..., on_failure=cleanup_fn)
does not trigger the function for cancellation. Oh and I'm using Prefect 1.0.
1
b

Bianca Hoch

01/03/2023, 11:21 PM
Hi Olivér, can you share a bit more about your use case? A short description of your flow, the cleanup function, and how you are leveraging the clean up function would be helpful.
Also, if the function is intended to clean up resources, I'd recommend checking out this article.
o

Olivér Atanaszov

01/04/2023, 9:49 AM
Hi, as mentioned I'm using prefect 1
For example, I might run some jobs on a Kubernetes cluster using individual flow runs, which run using
KubernetesRun
on different nodes:
def cleanup(flow, state):
   # cleanup resources in K8s
   # kubectl delete jobs --all --namespace $USER

with Flow("myflow", ..., on_failure=cleanup) a flow:
   # do some work, e.g. run various child flow runs on K8s
   create_flow_run("kubernetes_job")
   create_flow_run("kubernetes_job")
   # ...
Now, if I cancel the "main" flow named "myflow" in this example, I would like to run the
cleanup
function to make sure all resources are released. Does this make sense?
b

Bianca Hoch

01/09/2023, 9:56 PM
Hi Oliver, apologies for the delay here. Have you tried encapsulating your cleanup function with the
@resource_manager
decorator? It allows for custom setup and cleanup of cloud resources like kubernetes deployments (more docs on that here, if needed). There is an example of how to use that in the article posted above as well.
I imagine you could leverage that cleanup task with a state handler as well (although I haven't tried it personally, just a thought). The
on_failure
flow parameter will be called when a flow enters a failed state, and not a
Cancelled
state. You could try creating your
resource_manager
task, passing it to a state handler function which checks to see if the the flow
is_finished
(finished captures the
Cancelled
state), and then passing the state handler to the flow.
Ah, okay, I found this article here which gives a good summary for taking action on flow run cancellation. TL;DR "reacting to a Cancelled state allows to send notifications on a flow run cancellation or to shut down remote execution separate from the agent, but it doesn’t allow to clean up the agent’s / flow run’s compute resources."
o

Olivér Atanaszov

01/10/2023, 3:29 PM
thanks for the help 🙂
🚀 1
1