Hello everyone, we are running several flows in Prefect v1.
We are using k8s with Dask workers. We recently noticed an issue where some tasks are executed multiple times even though first execution succeeded. We believe this happens (mostly) when a Dask worker dies.
We also found an open issue that confirms the conditions for this to happen: https://github.com/PrefectHQ/prefect/issues/5485
I understand there is less appetite to working on Prefect1 with v2 now out and being utilized but since we are not ready to make the migration right now, I was hoping someone can give us some more guidance on how we can possibly mitigate this issue. Thank you!!