Morning team! 👋
Q: When a prefect flow is using DaskExecutor and is cancelled is there a way to force the workers to not finish the task they’re working on? What I’m seeing is that dask-workers are gracefully stopped when a flow is cancelled. For long running tasks it can be quite a while before the resources are freed.
k
Kevin Kho
09/14/2021, 2:12 PM
Hey @Alexander van Eck, the answer here is not really and the user has to manage that themselves because it’s very very hard to stop a Python process running on different compute.
See this StackOverflow post
a
Alexander van Eck
09/14/2021, 2:21 PM
Alright! Then I’ll have to look for that somewhere else - possibly we may be able to kill the container the worker is running in 🤔
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.