https://prefect.io logo
Join Slack
Powered by
This is going to sound stupid, but if I change a F...
# ask-community
d
This is going to sound stupid, but if I change a Flow state to cancelled or something similar, shouldn’t that Flow stop?
👀 1
d
Hey @Darragh! The 
Cancelled
state follows a “best attempt” design pattern. It’s extremely difficult to kill arbitrary python processes, especially ones that are running on shared Dask Clusters or the like. Here’s the PR where we added the “cancellation lite” functionality to Prefect Server (which I believe you’re using): https://github.com/PrefectHQ/prefect/pull/2535
We should have docs about this. I’ll open an issue to add them 👍
But basically, putting a Flow Run in a 
Cancelled
state may not stop the run immediately, but it should stop any new Task Runs from starting. Once 
Running
Task Runs enter a Finished state, the flow run should stop
d
Thanks for that Dylan, but my problem is how to cancel a currently running task? I have a task that ioterates over a bunch of files, and I spotted an error in the logs, so I went to cancel the Flow. -Flow moves to cancelled, but that task is still running
d
@Marvin open issue “Documentation for ‘Cancellation-lite’ Functionality”
shaking angry fist 1
@Darragh that depends on how your flow is running. I believe you’re running in AWS Fargate, so I think you’ll have to: 1. Set the Flow Run + Task Run to a 
Cancelled
state 2. Interact with Fargate to delete the run infrastructure
For a Task Run that is currently running
We’re looking to improve & expand this functionality as much as we can. If you have an idea on how we could specifically stop a Task Run / Flow Run in AWS Fargate, we’d love to hear it!
@Marvin open “Documentation for ‘Cancellation-lite’ Functionality”
m
d
Ah. So for an external cancel/state change, I have to make sure I kill everything manually?
d
You don’t have to if it’s okay for the Task Runs that are currently in a 
Running
state to resolve on their own
Only if you need them to stop immediately
Oh I see
I suggested it because Prefect Cloud has the Lazarus process which will revive a flow run it loses communication with, I don’t believe Prefect Server has that feature
You actually should be okay
d
How so? I’m seeing my Task continue with the Flow cancelled, not sure what you mean by I should be ok?
d
Sorry, let me clarify: if you manually delete your Flows execution architecture (in this case, the AWS Fargate infratructure) Prefect Cloud would actually attempt to re-start the flow run. Prefect Server will not
If you want the currently 
Running
Task Run to stop, you’ll need to kill its execution infrastructure manually
d
Ok. Thanks for that. I think this one could be a fairly big problem for us. I’ll check it out and get back to you
d
If you want the Flow Run to stop after the currently 
Running
Task Runs enter finished states, then setting the Flow Run to 
Cancelled
will do the trick
We’re looking to improve & expand this functionality as much as we can. If you have an idea on how we could specifically stop a Task Run / Flow Run in AWS Fargate, we’d love to hear it!
p
As a workaround, is there a way for a task to check if it should continue running? I realize it can be tricky but I am wondering if there an elegant way that long-running tasks could check periodically if they should continue running.
d
That is definitely possible!
Prefect has the concept of Task Looping
@Pedro Machado At the beginning of each loop, a task run could check an external condition (including its own start time to calculate a duration) and move to a different state based on that condition
Using signals
p
Is there something in prefect server/cloud that the task can check to see if it's supposed to continue?
d
That depends on what you mean by “supposed”
Can you give me an example?
If it’s based on information that Prefect Cloud has (metadata about the Flow Run) the probably
There’s also the Paused state for manual intervention/approvals
p
In the example above, can the task somehow find out that the Flow was manually set to 
Cancelled
or is there a way to send a signal to a task that is already running to tell it to stop? This would not work if the task is hung but if it's running as expected, it could decide that it needs to stop because the external signal was sent.
d
If you have Version Locking enabled for that flow, on every loop the task would check to make sure nothing else updated its state. So, if you manually set a Looping task to Cancelled, it would shut itself down after its current loop
@Darragh we’d love to hear your thoughts on https://github.com/PrefectHQ/prefect/issues/2771
d
@Dylan Commented! Thanks for pointing the issue out, great to be able to put the upvote on it!
d
Of course!
2 Views
Open in Slack
PreviousNext
Powered by