We often run into the case where we have a long-running task that we want to interrupt when we fail/cancel a flow. We had thought that perhaps setting the state of the task as failed would cause an interruption (in our case this means the worker pod on EKS should get terminated and then job should get terminated) but it does not - why is that the case ? and is there a way to kill the task without having to get access to the EKS cluster to kill the underlying pod ? More details on our setup: we are using prefect version 0.14.12, with prefect Cloud, a kubernetes agent and a dask-kubernetes execution environment on EKS.

We just call

cluster.close()

, so this also might be a dask-kubernetes issue.

will report back when I experiment with the type of tasks and perhaps share a somewhat reproducible example

@Jim Crist-Harif -sorry for the late reply but please find this sample flow - It just contains a single task that sleeps for a long-time

@task()
def long_sleep():
    time.sleep(10 * 60)

with Flow(self.name) as flow:
    long_sleep()

Setting the task to failed doesn’t interrupt the task or kill the worker pod - but instead waits until the sleep is complete Cancelling the flow on the other hand is more explicit about this behavior in the logs:

Stopping executor, waiting for 1 active tasks to complete

Is there a way from to interrupt this long running task from the cloud and kill the worker?

Environments are deprecated (in 0.14.0, we're now on 0.14.13).

thanks for shedding some light on this