Lukas N.
12/03/2021, 6:51 PMCloudFlowRunner
support graceful shutdown? We're running flows as Kubernetes jobs on EC2 spot instances, which get terminated from time to time. Let's say the job starts a pod a
on a node that terminates. Kubernetes will quickly reschedule it and spawn a pod b
, but b
does nothing because the state of the tasks is Running
, but they are not (a
is dead). We need to wait for the heartbeat to timeout for Prefect to reschedule it which takes a long time. Instead setting something like Pending
or even Failed
to tasks that are Running
when a SIGTERM is received would be nice.Kevin Kho
Kevin Kho
Lukas N.
12/03/2021, 7:29 PMLukas N.
12/03/2021, 7:32 PMKevin Kho
Kevin Kho
Kevin Kho
Anna Geller
Lukas N.
12/06/2021, 8:17 AM