Hi Everyone, is it possible to have a task that will only run if a flow fails? will i be able to use the state of a flow to run a task after the flow fails ?

Copying my response from thread above: If you want a python code to run that is not a task, put it in the flow state handler. If you want a task that runs if another task fails, use an 

all_failed

 trigger. But you can not chain a task to the flow failure because tasks need to be inside Flows

within my flow i am spinning up some clusters on the cloud. i use some tasks to submit the steps to the cluster and check the status of the step. based on the steps status i either submit other steps or terminate the cluster. it is possible that the cluster may not be shutdown so i want to run the task at the end of the flow irrespective of the previous tasks.

for the cleanup task you can use

all_finished

trigger (aka

always_run

). And in the case something went wrong with the task that creates a cluster, perhaps you can add retries and/or add a state handler that does something if this task failed?

my purpose of the task is to be doubly sure that the cluster is turned off. i think using

any_failed

would help since if any of my upstream task fail then i want to shut down the cluster. i have different cases within which i am already shutting down the cluster. this task is to just be doubly sure. if the cluster is turned off by any of the cases i have in the flow then running this task will not do anything since cluster is already turned off(which is fine). this is more for an edge case scenario where a cluster is spun up and i want to turn it off at the very end

oh, so you want to shutdown the cluster only if any task fails, rather than always when all upstream tasks finish? in that case, 100% correct,

any_failed

is what you need. I was thinking that you want to shutdown the cluster once all tasks are finished -

all_finished

would be better in that case