Hi everyone, I have Prefect in an ec2 instance, regularly snapshotted. Given that it is not possible in advance to predict how long a run will last, it is not possible to have a time frame with not runs to take a snapshot. This means the snapshots are most luckily to have a flow run in running state. Because of this, if the ec2 is restored, the flow run which has been snapshotted in running state will hang (it will stay in running state forever). It seems the flow run does not do anything. How do you suggest to deal with this?

you could snapshot programmatically, check with prefect api if anything is running first before actually snapshotting

I think that would bring to not have a snapshot due to running flow all day long except for some hours (which can have some running flow due to last variability)

What is it that you are trying to keep safe by snapshotting the instance? If there is data stored to disk could you offload that to block storage (s3) so that you can't loose it?

the snapshot is to automatically restore the instance in case any failure makes the autoscaling group kill the instance (e.g. if Prefect does not responds anymore in the healthcheck autoscaling group kills the instance and restore it)

Ahh ok that makes sense!