Hi <@ULVA73B9P> i suddenly got crash with error s...
# marvin-ai
s
Hi @Marvin i suddenly got crash with error sigkill signal, then prefect could not return failed
m
thought for 274.0 seconds
Thanks for the report — that’s frustrating. A few important points about signals and Prefect states in 3.x: - SIGKILL cannot be caught by any process. When a flow is killed by SIGKILL (commonly due to OOM), the process can’t send a final state to Prefect. Prefect will not mark this as Failed — by design it’s considered infrastructure failure and should be marked as Crashed. - Prefect detects unexpected termination via heartbeats. If heartbeats stop and no terminal state arrives within a window, the run should transition from Running → Crashed. If you saw the run remain “Running” after the SIGKILL, it usually means you need a “zombie flow” automation to mark it Crashed when heartbeats stop. What I recommend 1) Confirm the final state - Did the run end up in Crashed or stay stuck in Running? - If it stayed Running, add a “zombie flow” automation to mark it Crashed when heartbeats are missing. - In the UI, create an Automation: - Trigger: Flow run has not emitted a Heartbeat within ~90s - Action: Change flow run state to Crashed (and optionally Send Notification) - Reference: Detect zombie flows, Event triggers - Heartbeat cadence is ~30s by default; you can adjust with
PREFECT_RUNNER_HEARTBEAT_FREQUENCY
. 2) If you need “Failed” instead of “Crashed” - From Prefect’s perspective, SIGKILL is an infra issue, so the terminal state is Crashed. If your downstream expects Failed, add an automation: - Trigger: Flow run enters state Crashed - Action: Change flow run state to Failed - Or just handle Crashed the same as Failed in your alerting/pipelines. - State docs: Prefect states in 3.x 3) Investigate the cause of the SIGKILL - On Kubernetes: check the Pod’s termination reason (e.g., OOMKilled) and adjust requests/limits upward. - Local/VM: check dmesg/syslog for OOM killer messages and increase memory or reduce workload. Helpful CLI - You can inspect existing automations:
Copy code
prefect automation ls
(We verified this command exists.) A couple clarifying questions so I can be more specific: - Which Prefect version are you on (assuming 3.x unless you say otherwise)? - What infrastructure are you running on (Process, Docker, Kubernetes)? If K8s, do you see OOMKilled in the pod status? - Did the run end up Crashed, or did it remain Running with no final state? - Rough timestamp or a flow run ID we can reason about?
n
hi @seto do you have a stack trace?