Hi Adam, yes, that is one way of changing the flow run's state to
Cancelled
. Here is an example of how to accomplish this through the Orion client as well.
Bianca Hoch
09/28/2022, 2:17 PM
As a fair warning though, the most effective way of cancelling a flow run would be cancelling the job in the infrastructure. (ex: by killing the process on that machine or the K8s job).
Bianca Hoch
09/28/2022, 2:20 PM
I'll drop a link to this previous thread here, as Michael described the nuances of cancellation very well there.
a
Adam Eury
09/28/2022, 2:28 PM
Thank you @Bianca Hoch! Canceling the infrastructure our flow is running (dataproc/spark) was another idea we had, although it seemed a bit more complex since cancellation would be happening in a web app context that is separate from our data pipelines.
I saw this thread describe a way to handle state transitions such as cancellation from the prefect flow using the 1.0 SDK
https://discourse.prefect.io/t/is-it-possible-to-take-action-on-flow-run-cancellation/141
Does Prefect 2.0 currently support a similar mechanism? Cancellation via stopping the underlying infrastructure seems like it could make it challenging to do hooks like this, i.e. posting to slack.
Adam Eury
09/28/2022, 2:32 PM
If Prefect 2.0 supported those state hooks, it seems like we could stop the dataproc cluster in there maybe.
b
Bianca Hoch
09/29/2022, 6:13 PM
Hello Adam, apologies for the delayed response. I've raised this to the team to see if we already have an equivalent way of handling this in 2.0, or if this is something we will have in our roadmap. 😄
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.