Hi, it seems that when cancelling a flow run when there are tasks within a resource manager segment, the resource manager cleanup is not called, is that intentional (or can be enabled somewhere)?

Cancelling a flow run currently works as a hard-stop, killing all tasks as quickly as possible. There has been some thought on ways to ensure resource-managers are fully cleaned up even in the case of cancellation, but currently that's a bit tricky to ensure in all cases (there are several edge cases that are tricky to handle here).

That's because the resource manager's cleanup-task will be skipped as it is a downstream task of a skipped task. I haven't tested it, but I guess you can pass

skip_on_upstream_skip=False

to cleanup task's constructor.

The cleanup task already sets

skip_on_upstream_skip=False

- in a non-cancelled flow run it will always run if the resource-manager was successfully started. Cancellation is an edge case here, as it force-kills all tasks, leaving no time for the cleanup task to run.

Ah okay, yeah it makes sense to me.

Maybe there could be two separate ways to stop the flow, akin to docker's stop and kill? It makes sense to have a way to stop everything as soon as possible, but most of the time I would prefer to have a graceful shutdown, because now I have to clean up the resource manually after cancelling the flow.