When I cancel a flow in the UI, it stops further execution of tasks right? I got two questions/remarks:
• Is the active task killed in some way? Or will it wait until it completes?
• When I use a resource manager, I noticed that unfortunately the cleanup will not be called 😕 . Is there some way I can cancel my flow, but still cleanup resources? Maybe force it with a state handler, if state handlers still are triggered?
n
nicholas
01/08/2021, 3:25 PM
Hi @Joël Luijmes - in theory yes, the cancellation method makes a best-attempt to stop all running processes, including any tasks in a Running state. This isn't true in cases where you're using a shared Dask cluster, where instead of stopping running tasks, the flow runner will stop submitting tasks but allow any active tasks to run to completion.
nicholas
01/08/2021, 3:30 PM
However, because cancellation means attempting to send a
SIGINT
signal to the main thread, cleanup tasks will not be run.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.