Tim-Oliver
08/18/2022, 3:07 PMDaskExecutor
with cluster_class="dask_jobqueue.SLURMCluster"
. Everything works fine, unless SLURM kills the resources (e.g. due to time limit). Then the computation is naturally terminated, but in the Prefect cloud I still see the flow and task running. I did an experiment where SLURM kills the resouces after 2min, but the flow is still shown as running for 1h 38min and counting. I thought that the heartbeat option should kick in at some point to flag the flow as Failed
. Has anyone experience with a similar setup?Anna Geller
08/18/2022, 4:02 PM