https://prefect.io logo
Title
d

Darragh

06/11/2020, 5:34 PM
This is going to sound stupid, but if I change a Flow state to cancelled or something similar, shouldn’t that Flow stop?
👀 1
d

Dylan

06/11/2020, 5:38 PM
Hey @Darragh! The
Cancelled
state follows a “best attempt” design pattern. It’s extremely difficult to kill arbitrary python processes, especially ones that are running on shared Dask Clusters or the like. Here’s the PR where we added the “cancellation lite” functionality to Prefect Server (which I believe you’re using): https://github.com/PrefectHQ/prefect/pull/2535
We should have docs about this. I’ll open an issue to add them 👍
But basically, putting a Flow Run in a
Cancelled
state may not stop the run immediately, but it should stop any new Task Runs from starting. Once
Running
Task Runs enter a Finished state, the flow run should stop
d

Darragh

06/11/2020, 5:40 PM
Thanks for that Dylan, but my problem is how to cancel a currently running task? I have a task that ioterates over a bunch of files, and I spotted an error in the logs, so I went to cancel the Flow. -Flow moves to cancelled, but that task is still running
d

Dylan

06/11/2020, 5:41 PM
@Marvin open issue “Documentation for ‘Cancellation-lite’ Functionality”
😒haking-angry-fist: 1
@Darragh that depends on how your flow is running. I believe you’re running in AWS Fargate, so I think you’ll have to: 1. Set the Flow Run + Task Run to a
Cancelled
state 2. Interact with Fargate to delete the run infrastructure
For a Task Run that is currently running
We’re looking to improve & expand this functionality as much as we can. If you have an idea on how we could specifically stop a Task Run / Flow Run in AWS Fargate, we’d love to hear it!
@Marvin open “Documentation for ‘Cancellation-lite’ Functionality”
m
d

Darragh

06/11/2020, 5:50 PM
Ah. So for an external cancel/state change, I have to make sure I kill everything manually?
d

Dylan

06/11/2020, 5:52 PM
You don’t have to if it’s okay for the Task Runs that are currently in a
Running
state to resolve on their own
Only if you need them to stop immediately
Oh I see
I suggested it because Prefect Cloud has the Lazarus process which will revive a flow run it loses communication with, I don’t believe Prefect Server has that feature
You actually should be okay
d

Darragh

06/11/2020, 5:54 PM
How so? I’m seeing my Task continue with the Flow cancelled, not sure what you mean by I should be ok?
d

Dylan

06/11/2020, 5:57 PM
Sorry, let me clarify: if you manually delete your Flows execution architecture (in this case, the AWS Fargate infratructure) Prefect Cloud would actually attempt to re-start the flow run. Prefect Server will not
If you want the currently
Running
Task Run to stop, you’ll need to kill its execution infrastructure manually
d

Darragh

06/11/2020, 5:58 PM
Ok. Thanks for that. I think this one could be a fairly big problem for us. I’ll check it out and get back to you
d

Dylan

06/11/2020, 5:58 PM
If you want the Flow Run to stop after the currently
Running
Task Runs enter finished states, then setting the Flow Run to
Cancelled
will do the trick
We’re looking to improve & expand this functionality as much as we can. If you have an idea on how we could specifically stop a Task Run / Flow Run in AWS Fargate, we’d love to hear it!
p

Pedro Machado

06/11/2020, 8:57 PM
As a workaround, is there a way for a task to check if it should continue running? I realize it can be tricky but I am wondering if there an elegant way that long-running tasks could check periodically if they should continue running.
d

Dylan

06/11/2020, 8:57 PM
That is definitely possible!
Prefect has the concept of Task Looping
@Pedro Machado At the beginning of each loop, a task run could check an external condition (including its own start time to calculate a duration) and move to a different state based on that condition
Using signals
p

Pedro Machado

06/11/2020, 9:00 PM
Is there something in prefect server/cloud that the task can check to see if it's supposed to continue?
d

Dylan

06/11/2020, 9:01 PM
That depends on what you mean by “supposed”
Can you give me an example?
If it’s based on information that Prefect Cloud has (metadata about the Flow Run) the probably
There’s also the Paused state for manual intervention/approvals
p

Pedro Machado

06/11/2020, 9:02 PM
In the example above, can the task somehow find out that the Flow was manually set to
Cancelled
or is there a way to send a signal to a task that is already running to tell it to stop? This would not work if the task is hung but if it's running as expected, it could decide that it needs to stop because the external signal was sent.
d

Dylan

06/12/2020, 3:12 PM
If you have Version Locking enabled for that flow, on every loop the task would check to make sure nothing else updated its state. So, if you manually set a Looping task to Cancelled, it would shut itself down after its current loop
@Darragh we’d love to hear your thoughts on https://github.com/PrefectHQ/prefect/issues/2771
d

Darragh

06/12/2020, 7:28 PM
@Dylan Commented! Thanks for pointing the issue out, great to be able to put the upvote on it!
d

Dylan

06/12/2020, 7:28 PM
Of course!