This is going to sound stupid, but if I change a Flow state to cancelled or something similar, shouldn’t that Flow stop?

Hey @Darragh! The

Cancelled

state follows a “best attempt” design pattern. It’s extremely difficult to kill arbitrary python processes, especially ones that are running on shared Dask Clusters or the like. Here’s the PR where we added the “cancellation lite” functionality to Prefect Server (which I believe you’re using): https://github.com/PrefectHQ/prefect/pull/2535

Thanks for that Dylan, but my problem is how to cancel a currently running task? I have a task that ioterates over a bunch of files, and I spotted an error in the logs, so I went to cancel the Flow. -Flow moves to cancelled, but that task is still running

@Marvin open issue “Documentation for ‘Cancellation-lite’ Functionality”

Ah. So for an external cancel/state change, I have to make sure I kill everything manually?

You don’t have to if it’s okay for the Task Runs that are currently in a

Running

state to resolve on their own

How so? I’m seeing my Task continue with the Flow cancelled, not sure what you mean by I should be ok?

Sorry, let me clarify: if you manually delete your Flows execution architecture (in this case, the AWS Fargate infratructure) Prefect Cloud would actually attempt to re-start the flow run. Prefect Server will not

Ok. Thanks for that. I think this one could be a fairly big problem for us. I’ll check it out and get back to you

If you want the Flow Run to stop after the currently

Running

Task Runs enter finished states, then setting the Flow Run to

Cancelled

will do the trick

As a workaround, is there a way for a task to check if it should continue running? I realize it can be tricky but I am wondering if there an elegant way that long-running tasks could check periodically if they should continue running.

That is definitely possible!

Is there something in prefect server/cloud that the task can check to see if it's supposed to continue?

That depends on what you mean by “supposed”

In the example above, can the task somehow find out that the Flow was manually set to

Cancelled

or is there a way to send a signal to a task that is already running to tell it to stop? This would not work if the task is hung but if it's running as expected, it could decide that it needs to stop because the external signal was sent.

If you have Version Locking enabled for that flow, on every loop the task would check to make sure nothing else updated its state. So, if you manually set a Looping task to Cancelled, it would shut itself down after its current loop

@Dylan Commented! Thanks for pointing the issue out, great to be able to put the upvote on it!

Of course!